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Overview 


Ninth Grade Academies (NGAs) — also called Freshman Academies — have attracted national 
attention as a particularly intensive and promising approach for supporting a successful transi- 
tion for high school freshmen. An NGA is a self-contained learning community for ninth- 
graders that operates as a school within a school. NGAs have four core structural components: 
(1) a designated separate space within the high school, (2) a ninth-grade administrator who 
oversees the academy, (3) a faculty assigned to teach only ninth-grade students, and (4) teachers 
organized into interdisciplinary teams that have both students and a planning period in common. 
The theory of action behind NGAs is that when these components are employed together, they 
interact to create a more personalized learning environment where ninth-grade students feel less 
anonymous and more individually supported. This, in turn, should help students succeed in 
school and stay on track to high school graduation. NGAs have shown promising results when 
employed as part of a whole-school reform model, but in these cases schools have received ex- 
ternal support from a developer to create and sustain them. A growing number of schools and 
districts have been experimenting with NGAs on their own, but the little research that exists on 
their effectiveness is limited to anecdotal accounts. 

This study, which is based on a quasi-experimental research design, examines the effect of 
NGAs on students’ progress toward graduation, their academic achievement, and their behavior 
in several school districts in Florida. The sample for this study includes 27 high schools that 
created NGAs between 2001-2002 and 2006-2007, along with 16 comparison high schools that 
serve ninth-grade students with similar characteristics as students in the NGA schools. As con- 
text for understanding the impact findings, this study also looks at the extent to which the key 
features of the NGA model were implemented in the NGA schools in the study and how this 
differs from the structures and supports in the comparison schools. 

The key finding is that the NGAs in this study do not appear to have improved students’ aca- 
demic or behavioral outcomes (credit earning, state test scores, course marks, attendance, sus- 
pensions, or expulsions). The findings also suggest that it can be difficult for schools to fully 
implement the components of the NGA model without expert assistance: Three years after their 
creation, only half the NGAs in the study had all four structural components of the model in 
place. Nationally, school districts continue to create NGAs, and recent efforts to implement 
them have incorporated various enhancements that are intended to strengthen and improve their 
implementation, but little is known about their effectiveness. Because students’ experience in 
ninth grade is an important predictor of their future success, these efforts to create and improve 
NGAs should be examined in future studies. 
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Preface 


Although graduation rates have improved in recent years, far too many students still do not 
complete high school. Numerous students stray off the path to graduation when they enter high 
school, which can be a challenging transition for students both academically and socially. New 
freshmen must find their way around an unfamiliar and typically much larger school than they 
are used to. Unlike in middle school, students are expected to manage their own behavior, and 
they are held less accountable for class ahendance and the effort they put into their school work. 
As a result, the average student’s course performance declines in ninth grade. Preventing this 
collapse is crucial to helping students succeed. Important work by the University of Chicago 
Consortium for School Research and others has shown that students who pass their ninth-grade 
classes are more likely to graduate from high school and abend college. 

This report looks at the effect of an approach that has been adopted by many schools 
and districts in the country: Ninth Grade Academies (NGAs) — self-contained learning com- 
munities for ninth-graders that operate as small schools within larger high schools, with their 
own administrative leaders, faculty, space, and team organization. They strive to offer freshmen 
a more personalized, engaging, and responsive learning environment. 

Most of the high schools in the study, located in several Florida school districts, created 
their NGAs in 2005 or 2006. Using a quasi-experimental evaluation, the study found that these 
NGAs do not seem to have succeeded at improving their students’ academic and behavioral 
outcomes. But many of these schools were not able to implement the NGA model fully; only 
about half put in place all the model’s core components. An earlier implementation study con- 
ducted by MDRC in Broward County — one of the districts in this study — found that an even 
smaller proportion of schools implemented the components as strongly as intended. 

Despite these disappointing results, MDRC’s study should not be the last word. The 
Talent Development Secondary model, which begins with an NGA as a core component, has 
produced positive effects in other studies. Moreover, schools and districts across the country 
have continued to create NGAs, and recent efforts have combined NGAs with different kinds of 
enhancements to improve their implementation. Because students’ success in ninth grade is so 
important to their later success — and given how few interventions have been shown to help 
freshmen — there is a need for researchers and practitioners to continue to evaluate and 
strengthen NGAs so that they can fulfill their potential. 


Gordon L. Berlin 
President 
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Executive Summary 


The transition into high school is a volatile time for adolescents and a precarious point in the 
course of their education. Ninth-graders who successfully navigate this transition are far more 
likely to graduate from high school with their peers and attend college than those who experi- 
ence failure their freshman year. 1 Growing awareness of the importance of the first year of high 
school for future success has prompted schools and districts across the country to develop fonns 
of support and interventions designed specifically for ninth-graders. 

Ninth Grade Academies (NGAs) — also called Freshman Academies — have attracted 
national attention as a particularly intensive and promising approach for supporting a successful 
transition for high school freshmen. An NGA is a self-contained learning community for ninth- 
graders that operates as a school within a school. With its own administrative team, faculty, and 
space, an NGA is designed to offer ninth-graders a more personalized, engaging, and responsive 
learning environment and, by extension, to help students stay on track to high school gradua- 
tion. NGAs are a core element in several comprehensive high school reform models, such as 
High Schools That Work and Talent Development Secondary. In addition, a growing number of 
schools and districts have been experimenting with NGAs themselves, without external or spe- 
cialized assistance. 2 

This study, which is based on a quasi-experimental research design, examines the effect 
of NGAs on students’ progress toward graduation, their academic achievement, and their be- 
havior in several school districts in Florida. 3 As context for understanding the impact findings, 
this study also looks at the extent to which the key components of the model were implemented 
in the study schools that created an NGA and how these features differ from the structures and 
support services in a group of similar comparison high schools. 


What Are Ninth Grade Academies and Why Evaluate Them? 

An NGA is a self-contained small learning community in which a group of administrators and 
teachers work exclusively with ninth-grade students to create a school within a school that is 
responsive to the academic and social needs of incoming and repeating freshmen. NGAs have 
four core structural components: (1) a designated separate space in the high school, (2) a ninth- 
grade administrator who oversees the academy, (3) a faculty assigned to teach only ninth-grade 
students, and (4) teachers organized into interdisciplinary teams that have both students and a 


'Roderick, Kelley-Kemple, Johnson, and Beechum (2014). 

2 Kennelly and Monrad (2007); Chmelynski (2004); Kilanski, Smerdon, Legters, and Evan (2012). 

'This study is funded by a research grant from the U.S. Department of Education’s Institute of Education 
Sciences (Evaluation of State and Local Education Programs and Policies Grant #R305E090019). 
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planning period in common. In addition to these structural components, NGAs incorporate sup- 
portive practices that can include a “summer bridge” program for rising freshmen; a curriculum 
to help students who are behind catch up and close achievement gaps; extra academic help out- 
side of school hours; and a curriculum focused on career and college awareness. NGAs can also 
be paired with data-driven progress monitoring and positive behavior incentive systems. 4 

The theory of action behind NGAs is that when the core structural components of the 
NGA model are put in place together, they interact to foster a more personalized learning envi- 
ronment where ninth-grade students feel less anonymous and more individually supported. In a 
well-implemented NGA, ninth-grade students are located in a separate space, which promotes 
an environment where students and teachers can get to know each other and interact more regu- 
larly. NGA teachers work exclusively with ninth-grade students, so students have a consistent 
group of adults who are accountable for their success and are aware of their specific social and 
academic needs. In a well-implemented NGA, interdisciplinary teaching teams are important: 
Ninth-grade teachers from different content areas meet regularly to coordinate their course work 
and their instructional and behavioral management practices to better serve their students and 
maintain students’ academic effort and engagement. During these meetings, teachers also iden- 
tify students who need extra help and are especially at risk of dropping out, and strategize about 
how to connect them with specialized support services. A well-implemented NGA is also over- 
seen by a dedicated administrator, who coordinates all other components of the model so that 
ninth-grade students’ needs are met. Together, the structural components of the NGA model are 
intended to create a more personalized learning environment for students — one where students 
are less anonymous and more accountable, where they feel a greater sense of community, and 
where they receive individual help. Ideally, these changes in students’ ninth-grade experience 
should lead to improvements in their behavior and their academic achievement in ninth grade, 
and ultimately their progress toward high school graduation. 

Unfortunately, there is little rigorous empirical research on the effectiveness of NGAs. 
What research exists has focused on the effect of the comprehensive reform models that include 
NGAs, as opposed to the effect of NGAs in isolation. In general, these studies have found that 
NGAs — when implemented as part of a whole-school reform model — appear to improve stu- 
dent outcomes. A quasi-experimental study published by MDRC in 2005 investigated the im- 
pact of the Talent Development Secondary high school model — which includes NGAs as a 
core component — in five high schools in Philadelphia. Each of the high schools in the study 
received intensive, on-site support from facilitators at Johns Hopkins University, where the 
model was developed. The study found that the Talent Development Secondary model appeared 


4 Southem Regional Education Board (2002); Cook, Fowler, and Harris (2008). 
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to produce significant and substantial positive effects during students’ first year of high school 
in attendance, academic course credits earned, and promotion rates. 5 

Though promising, these findings reflect the effects of NGAs for schools that received 
external support from a developer to create and sustain their NGAs. Yet many schools and dis- 
tricts have been experimenting with NGAs by themselves, without external or specialized assis- 
tance. 6 Whether NGAs implemented in this context are effective is limited to anecdotal ac- 
counts that reflect mixed experiences and results. 7 

Therefore, an important and policy-relevant question is whether it is feasible to try to 
implement strong, effective NGAs in the absence of a whole-school reform approach and with- 
out external support from a developer. Accordingly, this study examines the effects and the 
characteristics of 27 NGAs that were created in several school districts in Florida from 2001 to 
2006, during their first three years of implementation. 


How Was the Effect of NGAs Evaluated in This Study? 

This study uses a comparative interrupted time series design (CITS) to evaluate the effect of 
NGAs. 8 In the CITS design, two groups of schools are studied: those that implemented an inter- 
vention (program schools) and those that did not (comparison schools). The first step in the 
CITS design is to look at the trend in student outcomes for the program and comparison schools 
during the school years before the intervention was launched (baseline trends). The second step 
is to gauge by how much the program and comparison schools “deviated” from their baseline 
trend after the intervention was launched. The two groups of schools are then compared not di- 
rectly on their outcomes but on deviations from their baseline trends. In the present study, the 
effect of NGAs is examined by looking at whether high schools that implemented NGAs expe- 
rienced larger and more positive deviations from trend than a group of matched comparison 
schools, with respect to their ninth-grade student outcomes (such as percentage of core credits 
earned by students). The CITS design is more rigorous than many other quasi-experimental de- 
signs because it combines data collected at several time points with data from a matched com- 
parison group. This combination makes it more plausible that estimated effects from a CITS 
design can be attributed to the effect of NGAs, because when the match is strong it can elimi- 
nate many alternative explanations for why deviations from trend would be different for the 


5 Kemple, Herlihy, and Smith (2005). 

6 For example, in the 2013-2014 school year, the DC Public Schools also decided to launch NGAs in nine 
high schools (District of Columbia Public Schools, 2013). 

7 Kennelly and Monrad (2007); Chmelynski (2004); Kilanski, Smerdon, Legters, and Evan (2012). 

8 For a discussion and history of CITS designs, see Shadish, Cook, and Campbell (2002). For a discussion 
of these designs in the context of education research, see Bloom (2003). 
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NGA schools and the comparison schools. 9 The CITS design has been shown to be able to re- 
produce the results of a randomized experiment in those circumstances when the criteria for a 
well-implemented design can be met (which they appear to have been in this study). 10 

The sample for this study includes 27 high schools in Florida that created NGAs. 
These NGAs were launched between 2001-2002 and 2006-2007 in eight urban school dis- 
tricts (Brevard, Broward, Duval, Hillsborough, Lee, Orange, Palm Beach, and Pinellas). The 
study sample also includes 16 comparison high schools whose trends in ninth-graders’ out- 
comes and whose school characteristics were similar to those of the NGA schools before 
NGA implementation. 11 Various sensitivity tests suggest that the comparison schools provide 
a credible reference point for what would have happened to the NGA schools had they not 
implemented the academies. This, in turn, increases this study’s ability to evaluate whether 
the NGAs caused any changes in ninth-grade student outcomes. 


Were the NGA Schools Able to Implement the Structural Features 
and Support Services of the NGA Model? 

This study examines the extent to which the 27 high schools were able to put in place the com- 
ponents of the NGA model. As discussed earlier, some components are not unique to the NGA 
model, so this study also examines whether elements of the NGA model were being used in 
some of the comparison schools as well. Information for this analysis comes from a survey that 
was sent to the administrators of all high schools in the eight study districts in spring 2011, ask- 
ing them to report on whether they were implementing the different structural components and 
support services associated with an NGA, and if so, in what year they started implementing 
them. The key findings are as follows: 

• Many of the NGAs in the study were unable to implement all four struc- 
tural components. Three years after their creation, only half the NGAs in 
the study had implemented all four components of the NGA model. 

• Some of the features of the NGA model were also part of the ninth-grade 
experience in the comparison schools. The comparison high schools in this 
study, for example, provide different kinds of support services to their ninth- 
grade students, and a substantial percentage of them have a dedicated ninth- 
grade faculty. 


9 Corrin and Cook (1998). 

10 St. Clair, Cook, and Hallberg (2014); Somers, Zhu, Jacob, and Bloom (2013); Fretheim et al. (2013). 

11 One comparison school was selected for each NGA school. However, the total number of unique com- 
parison schools is less than the total number of NGA schools, because some comparison schools were selected 
as the match for more than one NGA school. Statistical hypothesis tests are adjusted to account for the fact that 
some comparison schools are included in the analysis multiple times. 
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• The main difference between the NGAs and the comparison schools is 
with respect to space, leadership, and teaching teams. The NGAs in the 
study are much more likely than the comparison schools to report having a 
dedicated space for ninth-grade students, a dedicated administrator watching 
over the ninth grade, and teaching teams. 

An important finding to emerge from this study is that some of the features of the NGA 
model, particularly support services and dedicated faculty, are also part of the ninth-grade expe- 
rience in the comparison schools. This practical reality confirms that what distinguishes the 
NGA model is not its individual components, but rather the idea that when the components are 
put in place together, they should interact to promote an environment where adults make a con- 
certed effort to be aware of and responsive to student needs. Unfortunately, it was not possible 
in this study to measure how well the components of the NGA model were implemented in the 
study schools — only whether the components were present or absent . 12 Therefore, even though 
the NGA schools and the comparison schools do appear to differ structurally, it is unclear 
whether these differences were sufficient to have created a more personalized learning envi- 
ronment for ninth-grade students in the NGA schools. 


Did the NGAs in the Study Improve Student Outcomes? 

This study next examines whether the NGAs were able to achieve their ultimate goal: to im- 
prove ninth-grade student outcomes. Outcomes in three domains are explored: school pro- 
gress (as measured by credits earned in core subject areas ), 13 academic achievement (as 
measured by students’ proficiency on state tests in English language arts and math, and their 
grade point average in core subject areas), and student behavior (as measured by average at- 
tendance rates, in- and out-of-school suspensions, and expulsions). These outcomes were 
measured using student-level data for multiple cohorts of ninth-grade students, provided by 
the Florida Department of Education. A ninth-grade cohort is defined as a group of students 
who enter a high school in the same school year. The effect of NGAs is estimated for the first 
three cohorts of ninth-grade students to enroll in the study schools, corresponding to the first 
three years of NGA implementation. Table ES.l summarizes the findings for the third year of 
implementation. 


12 Measuring implementation quality would have required conducting site visits to observe the teachers and 
ninth-grade students in the NGAs. This was not possible because the study is retrospective (it is based on a 
sample of ninth-grade students who were in the NGAs before the start of this research study). 

°The core subject areas are English language arts, math, science, and social studies. 
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Table ES.l 


Estimated Effect of NGAs on Student Outcomes in Ninth Grade, 
Third Year of NGA Implementation 


Outcome 

Deviations from Baseline Trend 

NGA Comparison Estimated 

Schools Schools Effect 

P- Value for 

Estimated 

Effect 

Progress toward graduation 

Credits earned in core subjects 
(as % needed to graduate) 

-3.24 ftt 

-1.92 

-1.31 

0.397 

Academic achievement 

Proficient on ELA state test (%) 

2.47 ft 

0.82 

1.65 

0.409 

Proficient on math state test (%) 

-3.44 ff 

-1.29 

-2.15 

0.307 

GPA in core subject areas (out of 4.0) 

-0.06 

-0.09 f 

0.03 

0.619 

Behavior (%) 

Attendance rate 

-2.14 ftt 

-1.76 t 

-0.38 

0.748 

Received an in-school suspension 

-1.17 

2.04 

-3.21 

0.448 

Received an out-of-school suspension 

2.49 

4.62 ftt 

-2.13 

0.423 

Were expelled 

-0.02 

0.05 

-0.06 

0.155 

Number of schools (total = 43) 

27 

16 




SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts; GPA = grade point average. 

The findings in this table represent the estimated effect of NGAs for the third cohort of students who 
entered the study schools after the NGAs were launched. A ninth-grade cohort is defined as a group of 
students who enter a high school in the same school year. Ninth-grade outcomes are measured in students' 
first year of high school. 

The values in the columns labeled "NGA Schools" and "Comparison Schools" are the estimated 
deviations from baseline trend. Rounding may cause slight discrepancies in calculating sums and 
differences. 

A two-tailed t-test was applied to estimated deviations and estimated differences between NGA schools 
and comparison schools. The statistical significance of estimated deviations is indicated as: f f f = 1 percent; 
f f = 5 percent; f = 10 percent. The statistical significance of estimated effects is indicated as: *** = 1 
percent; ** = 5 percent; * = 10 percent. 


• Overall, there is no conclusive evidence that the NGAs in this study im- 
proved ninth-grade students’ outcomes. 

After the NGAs were created, ninth-grade students in the NGA schools actually per- 
formed worse than predicted (relative to their schools’ baseline trends) on a range of different 
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academic and behavioral outcomes. So did students in the comparison schools. This suggests 
that external factors unmeasured in this study (such as a new statewide accountability system) 
may have caused a general decline in student outcomes in all the study schools, and that creat- 
ing NGAs did not help schools avoid this decline. 

For example, Figure ES.l shows the trend in credits earned in core subject areas in 
ninth grade (as a percentage of core credits required for graduation). In the years before the 
NGAs were created, credit earning was on the rise in both the NGA schools and the comparison 
schools. But after the NGAs were launched, the NGA schools’ deviations from their baseline 
trend were statistically negative for this outcome, which indicates that ninth-grade students in 
the follow-up cohorts earned fewer credits than would have been predicted based on earlier co- 
horts’ credit earning. Ninth-grade students in the comparison schools also performed worse than 
expected. Although the NGA schools deviated from their baseline trend by a slightly greater 
(more negative) amount than the comparison schools on the percentage of credits earned, the 
difference in their deviations from trend is not statistically significant (estimated effect = -1.31 
percentage points and p-value = 0.397 in the third year of implementation). The conclusion is 
that the NGA schools did not improve the percentage of credits earned by ninth-grade students, 
whether in absolute terms or relative to the comparison schools. 

• Nor is there any conclusive evidence that the subgroup of NGA schools 
that implemented all four core structural components had an effect on 
ninth-grade students’ outcomes. 

The effect of NGAs was separately examined for the 14 NGA schools that implement- 
ed all four core structural components during the follow-up period (dedicated space, dedicated 
administrator, dedicated faculty, and teaching teams). The hypothesis is that by doing so, these 
schools were able to set up an organizational structure more conducive to creating a personal- 
ized learning environment for ninth-grade students. However, estimated effects on student out- 
comes for these 14 NGA schools are similar to the results for the full sample. 


Discussion 

The findings from this study suggest that creating effective Ninth Grade Academies can be 
difficult. The NGAs in this study do not appear to have improved ninth-grade students’ pro- 
gress toward graduation, their academic achievement, or their behavioral outcomes. Even 
among the subgroup of NGA schools that implemented all four core structural components, 
student outcomes do not seem to have improved. One hypothesis for this pattern of results is 
that the core NGA components may not have been implemented as intended in the study 
schools, and that as a result, staff members were not able to create the anticipated personal- 
ized ninth-grade experience. 
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Figure ES.l 


Baseline Trends and Deviations from Baseline Trend in the Percentage of 
Core Credits Earned in Ninth Grade 



Ninth-grade cohort 


SOURCE: Student records from the Florida Department of Education. 

NOTE: The estimated effect of NGA schools is equal to the estimated deviation from baseline trend for the NGA schools minus the estimated deviation from 
baseline trend for the comparison schools. The estimated effect of NGA schools shown in this figure is not statistically significant at the 10 percent level. 



In recent years, districts and comprehensive refonn models using NGAs have been 
strengthening their academies with additional elements aimed at improving their implementa- 
tion — such as monitoring systems to identify students at risk of dropping out, mentoring by 
near peers, and communities of practice that bring together NGA staff members to receive train- 
ing, share their experiences, and leam from each other. Because students’ experience in ninth 
grade is an important predictor of their future success, these efforts to create and improve NGAs 
should be examined in future studies. In order to provide the most rigorous evidence of their 
effectiveness, it would also be of benefit to the field for researchers to partner with districts that 
would be willing to randomly select which high schools will implement new NGAs; as of yet 
all studies of NGAs have been based on quasi-experimental research designs. Future studies 
should also measure how well the NGA components are being implemented and whether they 
create a more personalized learning environment, so that the NGA theory of action can be fully 
understood and tested. 
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Chapter 1 

Introduction 


The transition into high school is a volatile time for adolescents and a precarious point in the 
course of their education. Ninth-graders who successfully navigate this transition — and pass 
their ninth-grade classes — are far more likely to graduate from high school with their peers and 
attend college than those who experience course failure in their freshmen year . 1 

Unfortunately, the traditional structure and organization of a high school does not pro- 
mote a successful ninth-grade transition. In most high schools, teachers assume that students 
should manage their own behavior. Ninth-grade students are given more autonomy than in mid- 
dle school, and academic effort is framed as an option rather than an obligation . 2 Because high 
school teachers are organized by department (math, English language arts, etc.), they work with 
students across all grade levels, which makes it more challenging for them to pay close attention 
to their ninth-grade students’ academic effort, engagement, and attendance. Faced with this new 
freedom and relative anonymity, the average student shows a decline in course performance 
after entering high school — not because the material is more difficult, but because the student 
exerts less academic effort and attends classes less regularly . 3 

Yet growing awareness of the importance of the first year of high school for future suc- 
cess has prompted schools and districts across the country to develop support services and in- 
terventions designed specifically for ninth-graders. Ninth Grade Academies (NGAs) — also 
called Freshman Academies — have attracted national attention as a particularly intensive and 
promising approach for supporting a successful transition for high school freshmen. An NGA is 
a self-contained learning community for ninth-graders that operates as a school within a school. 
With its own administrative team, faculty, and space, an NGA is designed to offer ninth-graders 
a more personalized, engaging, and responsive learning environment that is closer to what stu- 
dents would experience in middle school. This, in turn, should help students to better navigate 
the high school environment and stay on track to graduation. 

This study, which is based on a quasi-experimental research design, examines the effect 
of NGAs on students’ progress toward graduation, their academic achievement, and their be- 
havior in several school districts in Florida . 4 To provide context for understanding the impact 


'Roderick, Kelley-Kemple, Johnson, and Beechum (2014); Allensworth and Easton (2005); Allensworth 
and Easton (2007). 

2 Rosenkranz, de la Torre, Stevens, and Allensworth (2014). 

3 Rosenkranz, de la Torre, Stevens, and Allensworth (2014). 

4 This study is funded by a research grant from the U.S. Department of Education’s Institute of Education 
Sciences (Evaluation of State and Local Education Programs and Policies Grant #R305E090019). 
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findings, this study also looks at the extent to which the key features of the NGA model were 
implemented in the NGA schools in the study, and how these features differ from the structural 
characteristics and support services in schools that are not implementing NGAs. 

The remainder of this chapter provides further background on NGAs — including a 
summary of existing research on their effectiveness — and the research questions that guide this 
study. Chapter 2 discusses the analytical framework used in this report. Chapter 3 examines the 
implementation of the NGAs and the features offered in the non-NGA high schools. Chapter 4 
looks at the effect of the NGAs on ninth-grade student outcomes. Chapter 5 concludes by re- 
flecting on some of the key lessons that can be drawn from the study’s results. 


Ninth Grade Academies: Features and Theory of Action 

An NGA is a self-contained small learning community in which a group of administrators and 
teachers work exclusively with ninth-grade students to create a personalized school within a 
school that is responsive to the academic and social needs of incoming and repeating fresh- 
men. NGAs have four core structural components: (1) a designated separate space in the high 
school, (2) a ninth-grade administrator who oversees the academy, (3) faculty assigned to 
teach only ninth-grade students, and (4) teachers organized into interdisciplinary teams that 
have both students and a planning period in common. In addition to these structural compo- 
nents, NGAs also incorporate instructional and student support practices aimed at promoting 
success for ninth-grade students. Instructional support can include an interdisciplinary curric- 
ulum coordinated across ninth-grade classes; flexible scheduling that allows teachers to re- 
group students as needed to provide targeted help or skill development; a curriculum to help 
students who are behind catch up and close achievement gaps; and a curriculum focused on 
career and college awareness. NGAs may also support students with a “summer bridge” pro- 
gram for rising freshmen, extra help in core academic subjects, data-driven progress monitor- 
ing, and positive behavior incentive systems. 5 

The theory of action behind NGAs is that when the core structural components of the 
NGA model are implemented together, they interact to foster a more personalized learning envi- 
ronment where ninth-grade students feel less anonymous and more individually supported (see 
Figure 1.1). In a well-implemented NGA, ninth-grade students are located in a separate space, 
which promotes an environment where students and teachers can get to know each other and 
interact more regularly. NGA teachers work exclusively with ninth-grade students, so students 
have a consistent group of adults who are accountable for their success and are aware of their 
specific social and academic needs. Interdisciplinary teams of ninth-grade teachers, who share 


Southern Regional Education Board (2002); Cook, Fowler, and Harris (2008). 
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Figure 1.1 

Theory of Action for Ninth-Grade Academies 
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students, meet regularly to coordinate their course work and their instructional and behavior 
management practices so that they can respond to the needs of their students and help them 
maintain their academic effort and engagement. During these meetings, teachers also identify 
students who need extra help and are especially at risk academically and develop strategies to 
connect them with specialized support. A well-implemented NGA is also overseen by a dedi- 
cated administrator , who coordinates all other components of the model so that ninth-grade 
students’ needs are met. Together, the structural components of the NGA model are intended to 
create a more personalized learning environment. (Box 1.1 presents a summary of what this en- 
vironment involves.) Ideally, these changes in students’ ninth-grade experience should lead to 

Box 1.1 

What Is a Personalized Learning Environment and 
How Do NGAs Foster It? 

Ninth Grade Academies (NGAs) are intended to create a more personalized learning envi- 
ronment for ninth-grade students than in a typical high school. What are the elements of this 
environment? 

• Less anonymity and greater accountability. Students feel like they are part of a com- 
munity of learners and that they are accountable for their behavior and academic 
perfomiance. 

• Individual help. Students receive individually tailored help and support services that are 
responsive to their specific academic and social needs. 

When the core structural components of an NGA are well implemented, they can help create 
this kind of environment for ninth-grade students. 

• Dedicated space. Ninth-grade classes are located in a separate space. This addresses ano- 
nymity and accountability by creating a communal environment where teachers and stu- 
dents know each other better and share common goals. 

• Dedicated faculty. Because teachers work exclusively with ninth-graders, they are more 
aware of the specific needs of their students. 

• Interdisciplinary teaching teams. Teachers share a group of students and meet with each 
other regularly to discuss the challenges faced by specific students and to develop strate- 
gies for helping them. 

• Dedicated administrator. The ninth-grade administrator oversees the NGA and coordi- 
nates staff efforts to be aware of and responsive to students’ needs. 




improvements in their behavior and their academic achievement in ninth grade, and ultimately 
their progress toward high school graduation. It is worth noting that some components of the 
NGA model (like dedicated faculty and teaching teams) can be used by schools independent of 
creating an NGA. Therefore, the particular value (and anticipated effects) of the NGA model 
stem from the interaction among its components and the personalized learning environment that 
it is intended to create. 

NGAs are a core element in several comprehensive high school reform models. For ex- 
ample, as part of the High Schools That Work (HSTW) model, an initiative developed by the 
Southern Regional Education Board, many high schools designate a separate part of the build- 
ing for ninth-grade students, and ninth-grade teachers have common planning time where they 
assess students and arrange interventions for students who are struggling. 6 NGAs are also a crit- 
ical element of Talent Development Secondary (TDS), which is a whole-school education re- 
form developed by Johns Hopkins University that combines school restructuring with curricular 
and instructional reforms, as well as extended learning time for students at risk of dropping out. 
Both HSTW and TDS offer on-site technical assistance to schools implementing these models. 

A growing number of schools and districts have been experimenting with NGAs them- 
selves, without external or special assistance. 7 Through the federal Small Learning Communi- 
ties grant program, for example, several school districts have received funding from the U.S. 
Department of Education to implement NGAs. 8 In the 2013-2014 school year, DC Public 
Schools launched NGAs in nine high schools using a combination of district and federal funds. 9 

Debates persist, however, about the design of NGAs, and in particular the wisdom of 
separating ninth-graders from upper-grade students. A key concern is that NGAs isolate ninth- 
graders and delay their full entry into high school. High schools in Sarasota, Florida, for exam- 
ple, discontinued their NGA initiative in part because administrators and parents perceived that 
ninth-graders were being forced to navigate two difficult transitions, one into the NGA and then 
a second into tenth grade. 10 

There are also questions about the feasibility of implementing NGAs. Simply putting in 
place the structural components of an NGA is not sufficient for creating a more personalized 
learning environment. School staff members must also make a concerted effort to work together 
to “activate” these components and take advantage of the opportunities afforded by the NGA 


6 Smith (2005). 

7 Kennelly and Monrad (2007); Chmelynski (2004); Kilanski, Smerdon, Legters, and Evan (2012). 

8 For example, among the 2010 district or state grantees of the U.S. Department of Education’s Small 
Learning Communities grant program, several of the 28 grantees planned to create an NGA or enhance an ex- 
isting NGA in their high schools (U.S. Department of Education, 2015). 

"’District of Columbia Public Schools (2013). 

10 Scott (2006). 
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structure. Yet, as content area specialists, high school teachers tend to interact more with teach- 
ers in their own departments, so interdisciplinary teaming and cooperation can be unfamiliar 
territory. Therefore, schools implementing NGAs may need to provide additional training and 
support to their ninth-grade teachers on how to work more effectively with each other across 
disciplines to meet the specific needs of ninth-grade students. External developers — because 
they can offer various additional support services and technical assistance — may be better 
equipped to help the school staff build collaborative relationships and systems that make the 
most of the NGA structure. For an individual school district that is trying to implement NGAs 
on its own, providing these additional staff support services could be challenging. This may ex- 
plain why some district practitioners report very positive experiences with NGAs, while others 
have abandoned the NGA concept because they found it too costly and difficult. 


Prior Research on NGAs and This Study’s Research Questions 

Given these debates, it is unfortunate that there is little rigorous empirical research on the effec- 
tiveness of NGAs. What research exists has focused on the effect of the comprehensive reform 
models that include NGAs, as opposed to the effect of NGAs in isolation. In general, these stud- 
ies have found that NGAs — when implemented as part of a whole-school reform model — 
appear to improve student outcomes. A quasi-experimental study published by MDRC in 2005 
investigated the impact of the TDS high school model — which includes NGAs as a core com- 
ponent — in five high schools in Philadelphia. Each of the high schools in the study received 
intensive, on-site support from facilitators at Johns Hopkins University, where the model was 
developed. The study found that the TDS high school model appeared to produce significant 
and substantial positive effects during students’ first year of high school in attendance, academic 
course credits earned, and promotion rates. Effects on credits earned and promotion rates also 
appeared to be sustained as students moved through high school. 11 Another nonexperimental 
study conducted by the North Carolina Department of Public Instruction examined the effect of 
NGAs implemented under a variety of whole-school reform models, including TDS and 
HSTW. The study found that ninth-grade students in NGA schools had lower nonpromotion 
and dropout rates when compared with students in similar schools across the state. The analysis 
was based on six years (2001-2007) of school summary data for 82 NGA schools and compara- 
ble high schools in the state. 12 

Though promising, these findings reflect the effects of NGAs for schools that received 
external support from a developer to create and sustain their NGA, and where the NGA is em- 
bedded within a larger reform model that makes a concerted effort to support ninth-grade 
students as they leave the NGA and make the transition into tenth grade. As noted earlier, how- 

n Kemple, Herlihy, and Smith (2005). 

12 Cook, Fowler, and Harris (2008). 
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ever, many schools and districts have initiated NGAs themselves, without (or with very little) 
external support. Whether or not NGAs implemented in this context are effective is limited to 
anecdotal accounts that reflect mixed experiences and results. 13 Given that districts and schools 
across the country continue to create NGAs, an important and policy-relevant question is 
whether it is feasible to try to implement strong and effective NGAs in the absence of a whole- 
school refonn approach and without external support from a developer. 

Accordingly, this study examines the effects and the characteristics of 27 NGAs that 
were created in several school districts in Florida from 200 1 to 2006. The following questions 
are addressed: 

1. To what extent were the key features of the NGA model implemented by the 
NGA schools in the study, and how do these features differ from the structural 
characteristics and support services in schools that are not implementing 
NGAs? 

2. Did the NGAs improve students’ school progress, their academic achieve- 
ment, and their behavior in ninth grade? 

The effect of the NGAs on student outcomes is examined using a comparative inter- 
rupted time series design. In this type of design, effects are assessed by looking at whether, after 
their NGAs were created, the average ninth-grade outcomes of schools that implemented NGAs 
“deviated” from their pre-NGA trends by a greater amount than a group of matched comparison 
schools with similar preintervention trends. This study also examines the extent to which, in 
practice, the NGAs in the study differed from their matched comparison schools with respect to 
their structural features and the support services provided to students. Thus, this study provides 
an opportunity to examine both the characteristics and potential effects of NGAs in a realistic 
and policy-relevant context. 


13 Kennelly and Monrad (2007); Chmelynski (2004); Kilanski, Smerdon, Legters, and Evan (2012). 
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Chapter 2 

Study Design 


This chapter describes the analytical approach used in this study to evaluate the effect of Ninth 
Grade Academies (NGAs) on student outcomes. The first section provides an overview of the 
comparative interrupted time series (CITS) design, which is the quasi-experimental research 
design on which this study relies. The second section discusses the data sources that are used to 
measure the student outcomes and school characteristics of interest. The third section describes 
the NGA schools and the comparison schools included in this study, which are similar to each 
other with respect to the characteristics and baseline outcomes of their students. 


The Comparative Interrupted Time Series Design 

In this study, the effect of NGAs on student outcomes is evaluated using a CITS design, which 
is a quasi-experimental design that lends itself well to exploring the effect of school-wide inter- 
ventions. CITS designs have been used for decades to evaluate interventions in areas such as 
epidemiology, political participation, substance abuse, advertising, and employment programs. 1 
In education, CITS designs have been used to evaluate federal policies like No Child Left 
Behind and the effect of whole-school reform models. 2 Studies have shown that a well- 
implemented CITS design can, in some circumstances, reproduce the results of a randomized 
experiment. 3 

In the CITS design, program impacts are evaluated by looking at whether schools that 
implemented an intervention (in this case, NGAs) “deviated” from their baseline trends by a 
greater amount than a group of similar comparison schools. Figure 2.1 illustrates how the CITS 
design is applied in this study, using hypothetical data on students’ progress toward graduation 
as measured by the percentage of required core subject credits that they have earned in ninth 
grade. The steps in the CITS design are as follows: 

• Mean outcomes by cohort. First, student-level data are used to create a time 
series that represents the percentage of core credits that were earned by con- 
secutive cohorts of ninth-grade students in the NGA schools (black dots) and 
the comparison schools (white dots). A ninth-grade cohort is defined as a 


'Bloom and Riccio (2005); Ballart and Riba (1995); Campbell and Ross (1968); Mulford, Ledolter, and 
Fitzgerald (1992). For a discussion and history of CITS designs, see Shadish, Cook, and Campbell (2002). For 
a discussion of these designs in the context of education research, see Bloom (2003). 

2 Dee and Jacob (2011); Wong, Cook, and Steiner (2009); Kemple, Herlihy, and Smith (2005). 

3 St. Clair, Cook, and Hallberg (2014); Somers, Zhu, Jacob, and Bloom (2013); Fretheim et al. (2013). 
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Figure 2.1 


Using a Comparative Interrupted Time Series Design to Estimate the Effect of NGAs on the 
Percentage of Core Credits Earned in Ninth Grade: A Hypothetical Example 
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group of students who enter a high school in the same school year. The co- 
horts who entered the study high schools before the NGAs were created are 
referred to as the “baseline cohorts” and the cohorts who entered after the 
NGAs were created are the “follow-up cohorts.” 

• Baseline trends. The next step is to estimate the pre-NGA trend in student 
outcomes, based on the baseline cohorts of ninth-grade students . 4 Baseline 
trends are separately estimated for the NGA schools in the study (solid black 
line) and the comparison schools (solid gray line). These trends are then pro- 
jected into the follow-up period, in order to predict what percentage of core 
credits the follow-up cohorts of students would have earned had their 
schools’ baseline trends persisted (dotted lines). 

• Estimated deviations from the baseline trend. The next step is to compare 
the percentage of core credits that the follow-up ninth-grade cohorts actually 
earned (the dots in the follow-up period) and what they would have earned 
had their schools’ pre-NGA trends persisted (the dotted lines). These “devia- 
tions from baseline trend” are estimated for the NGA schools and the com- 
parison schools for each follow-up cohort. 

• Estimated effect. Finally, the effect of NGAs is estimated as the difference 
between the average deviation from baseline trend for the NGA schools 
and the average deviation from trend for the comparison schools. If the 
program is effective, then average deviations from trend for the NGA 
schools will be positive and greater than deviations for the comparison 
schools. The effect of NGAs is estimated for each of the first three cohorts 
of ninth-grade students to enroll in the NGAs (corresponding to the first 
three years of NGA implementation ). 5 

The CITS design is more rigorous than most other quasi-experimental designs be- 
cause it combines time series data with a matched comparison group. Together, these two de- 
sign elements make it more plausible that estimated effects from a CITS design can be 
attributed to the effect of NGAs, because they can eliminate most alternative explanations for 


4 A student’s cohort is determined based on the first school year in which he or she entered ninth grade. For 
example, a student is a member of the 2005-2006 cohort if he or she enrolled in ninth grade for the first time in 
the 2005-2006 school year. 

Considering that it may take several years for a school to have all the NGA components in place with the 
intended level of quality — especially without outside technical assistance — it is important to look at the ef- 
fect of NGAs for as many follow-up cohorts as possible. Given the data available for this study, the effect of 
NGAs can be examined up to and including the third year of implementation. 


11 



why deviations from trend would be different for the NGA schools and comparison schools. 6 
First, by using several years of preintervention (baseline) data, the CITS design eliminates the 
possibility that estimated effects are actually due to differences in preexisting trends between 
the NGA schools and the comparison schools. 7 Second, by including a comparison group, the 
CITS design also reduces the possibility that estimated program effects are confounded with 
the effect of a state or district reform launched at the same time as the program being studied. 8 
In a CITS design, the effect of other educational reforms or initiatives is captured by the com- 
parison schools’ deviations from their baseline trend, which is subtracted out when estimating 
the effect of the program. 

This illustrates an important assumption of the CITS design: The comparison schools’ 
deviations from their baseline trend are supposed to represent the deviations from trend that the 
NGA schools would have experienced had they not implemented the program (“the counterfac- 
tual”). To maximize the likelihood that this assumption holds, the NGA schools in this study 
were matched to comparison schools with similar baseline trends in the outcomes of the ninth- 
grade students that they served. Matching on the outcomes of earlier cohorts of ninth-grade stu- 
dents makes it more probable that the NGA schools and the comparison schools were motivated 
and affected by the same contextual factors. The selection of comparison schools is discussed in 
greater detail later in this chapter. 


Measures and Data Sources 

This study looks at the effect of NGAs on student outcomes in three domains: school progress 
(as measured by the number of credits earned in core subject areas in ninth grade, as a percent- 
age of total core credits required for graduation); 9 academic achievement (as measured by stu- 
dents’ proficiency on state tests in English language arts [ELA] and math, and their grade point 
average [GPA] in core subject areas); and student behavior (as measured by average attendance 
rates, in- and out-of-school suspensions, and expulsions). These outcomes — which correspond 
to the short-term outcomes in the NGA theory of action (Figure 1.1) — are measured at the end 
of students’ ninth-grade year. 10 

Looking at the effect of NGAs on multiple outcomes increases the risk of mistakenly 
concluding that NGAs had a statistically significant effect on an outcome when in fact they did 


6 Corrin and Cook (1998). 

7 This is also referred to as maturation bias (Shadish, Cook, and Campbell, 2002). 

8 This is also referred to as historical bias (Shadish, Cook, and Campbell, 2002). 

9 The core subject areas are English language arts, math, science, and social studies. 

1 "Promotion to tenth grade is not used as an outcome in this study because there is variation in grade pro- 
motion policies across the school districts in the study. 
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not. 11 To reduce this risk, conclusions about NGAs’ effectiveness in this study are based only on 
whether NGAs have an effect on core credits earned (the primary outcome). This outcome is 
primary because passing courses in ninth grade is a key predictor of high school graduation. 12 
The effect of NGAs on the other outcomes is also examined, but only as a way of better under- 
standing effects on credits earned. 

All student outcomes were measured using student-level data for multiple cohorts of 
ninth-grade students provided by the Florida Department of Education (see Table 2.1 for further 
information). These data were obtained for the 1997-1998 to 2008-2009 ninth-grade cohorts for 
all high schools in eight urban school districts in Florida (Brevard, Broward, Duval, Hills- 
borough, Lee, Orange, Palm Beach, and Pinellas), which are referred to as the “study dis- 
tricts.” 13 Given available data, it is possible to look at the effect of the NGAs for the first three 
cohorts of ninth-grade students who enrolled in them. 

To understand the impact findings and put them in context, this study also looks at 
whether the schools in the study were able to put in place the core structural components and 
support services of the NGA model (the “intervention” in Figure 1.1). A survey was sent to the 
administrators of all high schools in the eight study districts in spring 2011. The survey asked 
administrators to report on whether they were implementing the different structural components 
and support services of an NGA, and if so, in what year they started implementing these com- 
ponents and supports. 14 Because the survey was sent to all schools, it can be used to examine 
not only whether the NGA schools in the study were able to implement the components of the 
model, but also to what extent these components differed from the characteristics and features 
of the comparison schools. 

An important limitation of the survey is that it captures only whether the NGA com- 
ponents and support services were in place in the study schools. It does not measure whether 
the NGA components were implemented well, or whether the intended staff and student inter- 
actions that characterize a strong NGA were actually happening. Nor does it measure whether 
the NGAs resulted in a more personalized learning environment for ninth-graders, which is a 
key intermediate step in the NGA theory of action (the “ninth-grade experience” column in 
the theory of action in Figure 1.1). Measuring implementation quality and personalization 
would have required administering a student survey in the study schools and conducting site 
visits to observe teachers and students. Collecting these data was not possible because this 


1 'This is called a Type 1 error. 

12 Roderick, Kelley-Kemple, Johnson, and Beechum (2014); Allensworth and Easton (2005); Allensworth 
and Easton (2007). 

13 See Appendix A for further information about response rates for these data. 

14 The average response rate on the administrator survey was 70 percent. 
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Table 2.1 


Sources for Ninth-Grade Student Outcomes and Measures of NGA Components 


Outcome or Measure 

Data Source 

9th-Grade Cohort School Years 

Prosress toward eraduation 

Credits earned in core subjects (as % needed 
to graduate) 3 

Student records 

1997-1998 to 2008-2009 

Academic achievement 

GPA in core subject areas (4-point scale) 

Student records 

1997-1998 to 2008-2009 

Proficient on state ELA assessment 

Student records 

2000-2001 to 2008-2009 

Proficient on state math assessment 

Student records 

2000-2001 to 2008-2009 

Behavior 

Attendance rate 

Student records 

1997-1998 to 2008-2009 

Received an in-school suspension 

Student records 

1997-1998 to 2008-2009 

Received an out-of-school suspension 

Student records 

1997-1998 to 2008-2009 

Were expelled 

Student records 

1997-1998 to 2008-2009 

Ninth-grade structures and support services 

Presence of structural NGA components 

Administrator 

Survey b 

1997-1998 to 2009-2010 

Curricular support 

Administrator 

Survey b 

1997-1998 to 2009-2010 

Student support 

Administrator 

Survey 1 ’ 

1997-1998 to 2009-2010 


NOTES: ELA = English language arts; GPA = grade point average. 

“Primary outcome. 

b The Administrator Survey was sent out in Spring 2011. Administrators were asked to recall when they 
started/stopped implementing different NGA features. 

study is retrospective — it is based on a sample of ninth-grade students who were in the NGA 
schools before this study began. 

Finally, the study uses the Common Core of Data, collected by the National Center for 
Education Statistics, to describe the structural and demographic characteristics of the high 
schools in the study. 
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The Study Sample 

In total, 27 NGA schools are included in this study. These NGAs were identified from the pool 
of high schools in the eight study districts that responded to the administrator survey, based on 
their responses about their ninth-grade structures and systems. 15 To be included as an NGA 
school in the study, a school had to meet several criteria: 

• NGA components. Having a dedicated ninth-grade administrator or space 
is a strong signal that a school considers ninth-graders to have unique needs 
that require special attention. However, implementing one of these core 
structural components on its own would not be sufficient: At a minimum, 
each would need to be paired with one other structural component in order 
to make ninth-graders’ experience substantially different from their experi- 
ence in a more traditional setting. Therefore, in this study, a school was de- 
fined as having created an NGA if it implemented two of the four core 
structural components of the NGA model (dedicated space, dedicated ad- 
ministrator, dedicated faculty, and teaching teams), 16 and at least one of 
these components was either a dedicated administrator or a dedicated 
space. However, in theory, implementing all four core components could 
transform the ninth-grade experience even more, so this study also exam- 
ines what percentage of schools were able to implement all four compo- 
nents (Chapter 3). In addition, as an exploratory analysis, the study looks at 
the effect of NGAs on student outcomes for the subgroup of schools that 
met this stricter criterion (Chapter 4). 

• NGA start year. To be included in the study sample, a school must have 
created its NGA between 2001-2002 and 2006-2007, ensuring that student 
outcomes data were available for at least four baseline cohorts and three 
follow-up cohorts of ninth-grade students for use in the CITS analysis. 17 


15 The characteristics of the NGA schools in this study are quite similar to those of all schools in the study 
districts (see Table 2.2). Therefore, the survey respondents appear to be fairly representative of the average 
high school in the study districts. 

16 The teaching team for an NGA should be interdisciplinary, but for the purposes of defining NGAs, it 
was not always possible to determine from the survey whether the team was interdisciplinary, because of miss- 
ing data on the related survey item. Therefore, a school with any type of teaching team is considered to have 
implemented this component. 

17 As noted earlier, data were available for the 1997-1998 to 2008-2009 ninth-grade cohorts. Therefore, a 
school had to have launched its NGA between 2001-2002 and 2006-2007 to have four years of preintervention 
data and three years of follow-up data. 
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• Data availability. The school had to have complete data on the outcomes of 
interest for all its baseline and follow-up cohorts. 

• School characteristics. The school had to be a regular nonmagnet public 
school with a grade 9-12 configuration (so that the ninth grade represents a 
change in school for students). 

As Table 2.2 indicates, the majority of the NGAs in the study were launched in 2005- 
2006 and 2006-2007, and the study districts with the most NGA schools are Broward (10), 
Duval (5), and Pinellas (5). 18 Relative to the study districts and the state, the 27 NGA schools in 
this study and their students are somewhat more academically and economically disadvantaged, 
as shown in Table 2.3. Compared with all ninth-grade students in the study districts, the stu- 
dents in the NGA schools in this study were less likely to be proficient on the state reading and 
math tests in eighth grade before they entered high school, and more likely to be eligible for free 
or reduced-price lunch. Relative to other high schools in the state and the study districts, the 
high schools in which the NGAs in this study are housed are more likely to be in urban areas, to 
be eligible for Title I funds, 19 and to have a higher proportion of black students. 

The sample for this study also includes a group of 16 comparison schools. These 
comparison schools were selected from among the 3 1 schools that did not meet the study’s 
operational definition for an NGA (“non-NGA schools”) during the study period and that had 
outcome data for all student cohorts in the study design. 20 Each of the 27 NGA schools was 
matched to the non-NGA school in the comparison pool with the most similar ninth-grade 
outcomes and characteristics in the school years before NGA implementation. 21 Because the 
pool of non-NGA schools from which to choose was small, matching was conducted “with 


18 The NGA “start year” is the year that the school “started supporting ninth-graders with structures, re- 
sources, and activities that constitute a significant departure from traditional practice.” If a school did not report 
which year this happened — or if one of the four core components was implemented more than two years be- 
fore the self-reported start year — then the start year of the NGA is the year when the school first reported hav- 
ing a dedicated administrator or a dedicated space. 

l9 Title 1 is the federal funding stream designated for schools serving low-income students. 

20 The pool of non-NGA schools is small (3 1 schools) because not all schools in the study districts returned 
the administrator survey (30 percent did not respond) and because some schools that did return a completed 
survey had to be excluded for lack of a sufficiently long time series of data. 

21 Given the small number of non-NGAs available for matching, comparison schools were chosen based 
on a limited number of matching characteristics, including the following ninth-grade outcomes: baseline trends 
in core credits earned (the primary outcome), baseline means in GPA in core subjects, attendance rates, and 
retention rates. State test scores and schools’ demographic characteristics were not included in the matching, 
because it was not possible to find comparison schools that were similar to the NGA schools with respect to 
both these characteristics and the other (more important) student outcomes. 


16 



Table 2.2 


NGAs in the Study by District and Start Year 



Total 


First School Year of the NGA 


School District 

Number of 

NGA Schools 

2001- 

2002 

2002- 

2003 

2003- 

2004 

2004- 

2005 

2005- 

2006 

2006- 

2007 

Brevard 

1 





1 


Broward 

10 

1 


1 


4 

4 

Duval 

5 





2 

3 

Hillsborough 

1 





1 


Lee 

2 





1 

1 

Orange 

2 




1 

1 


Palm Beach 

1 





1 


Pinellas 

5 




1 

3 

1 

Total 

27 

1 

0 

1 

2 

14 

9 


SOURCE: Administrator Survey. 

NOTE: A school is considered to have an NGA if it implemented two of the four 
core structural components of an NGA (dedicated space, dedicated administrator, 
dedicated faculty, team teaching), and at least one component is a dedicated 
administrator or dedicated space. The first year of the NGA is the year when a 
school reported that it "started supporting ninth graders with structures, resources, 
and activities that constitute a significant departure from traditional practice." If a 
school did not report a year when this happened — or if one of the four core 
components was implemented more than two years before the self-reported start 
year — then the start year of the NGA is the year when the school first reported 
having a dedicated administrator or a dedicated space. 


replacement,” which means that a comparison school could be chosen as the “best match” for 
more than one NGA school . 22 


22 The literature on matching also recommends that matching be conducted locally (in this case within a 
district). In this study, matching was conducted locally in two districts, but in the other six districts it was not 
possible to match locally because there were too few similar schools in the district. However, the trends in stu- 
dent outcomes in the eight study districts are similar, so matching outside of the district can still provide a cred- 
ible counterfactual. 
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Table 2.3 


Characteristics of NGA Schools in the Study Compared with Other High Schools 
in the Study Districts and the State, 2008-2009 School Year 



NGA Schools 

Study Districts 



in the 

All Survey 

All 

All High Schools 

Characteristic 

Study Sample 

Respondents 

High Schools 

in Florida 

School characteristics 

Eligible for free/reduced-price lunch 
(% of all students) 

41.3 

36.3 

37.5 

40.5 

Racial/ethnic composition 
(% of all students) 

Black 

36.7 

29.1 

29.5 

24.2 

Flispanic 

17.0 

18.6 

19.3 

19.7 

White 

39.5 

45.2 

44.4 

50.6 

Other 

6.8 

7.0 

6.8 

5.4 

Ninth-grade enrollment 

576.1 

592.1 

582.0 

505.7 

Pupil-teacher ratio (all grades) 

17.8 

18.3 

18.0 

17.8 

Title I status (% of schools) 

74.1 

51.9 

55.4 

55.3 

Location (% of schools) 

City 

96.3 

88.5 

89.2 

76.2 

Town 

0.0 

1.0 

0.7 

5.5 

Rural 

3.7 

10.6 

10.1 

18.3 

Characteristics of 9th-grade students (%) 

Eligible for free/reduced-price lunch 

49.6 

44.5 

45.4 

NA 

English language learners 

6.2 

5.6 

5.8 

NA 

Proficient on 8th-grade ELA state test 

50.9 

55.7 

53.5 

NA 

Proficient on 8th-grade math state test 

64.8 

69.3 

67.7 

NA 

Number of schools 

27 

104 

148 

365 


SOURCES: Common Core of Data and student records from the Florida Department of Education. 

NOTES: ELA = English language arts; NA = data not available. 

The study districts are Brevard, Broward, Duval, Flillsborough, Lee, Orange, Palm Beach, and Pinellas. Values 
for "all high schools" (last two columns) include all noncharter and nonmagnet regular public high schools that 
serve ninth- to twelfth-grade students. 
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The final study sample includes 27 NGA schools and 16 unique matched comparison 
high schools. 23 Given this sample size, the study has the ability to detect an effect of 2.3 per- 
centage points on core credits earned (the primary outcome) for the first cohort of ninth-grade 
students to enroll in the NGA schools, 3.1 percentage points for the second cohort, and 3.9 per- 
centage points for the third cohort. 24 This translates into about one-third to one-half of a credit. 

As explained earlier, in a CITS design, the NGA schools and the comparison schools 
should be as similar as possible in terms of the characteristics and outcomes of the students that 
they serve, as this increases the likelihood that the two groups of schools were motivated and 
affected by the same factors and that they faced similar contextual challenges. This, in turn, 
makes the comparison schools a more credible reference point for the NGA schools. 

Accordingly, the first panel of Table 2.4 looks at the outcomes of the ninth-grade stu- 
dents who were enrolled in the study schools in the school year before the creation of the 
NGAs. These results show that, before the NGAs were created, ninth-graders in both the NGA 
schools and the comparison schools were struggling academically. Students in both groups of 
schools had earned about 22 percent of the core credits required for graduation (indicating that 
they were below the 25 percent benchmark for being “on track” to graduate), 25 and only a third 
of students were proficient on the ninth-grade ELA state test. Students’ average GPA in core 
subjects was about 2.2, or just above a C. 

The second panel of Table 2.4 looks at the middle school outcomes of these same stu- 
dents for each group of schools. These results provide information on whether the NGA schools 
and the comparison schools served ninth-grade students who, as they entered high school, were 
similar in terms of their earlier academic perfonnance. As shown in this table, both groups of 
schools served students who struggled academically before they entered high school: About 43 
percent to 44 percent of students were proficient on the ELA state test in eighth grade, and 56 
percent were proficient on the math state test. The table also shows that the NGA schools and 
the comparison schools served students who entered high school with similar levels of econom- 
ic and language needs: About 5 percent to 6 percent of ninth-grade students were English 


23 As stated above, the total number of unique comparison schools is less than the number of NGA schools 
because some comparison schools were chosen as the match for more than one NGA. Standard errors and hy- 
pothesis tests are adjusted to account for the fact that some comparison schools are included in the analysis 
multiple times. See Appendix A for further information about the impact analysis model. 

24 In a CITS design, the minimum detectable effect (MDE) is larger for later follow-up cohorts, because 
there is greater uncertainty about trend projections. See Bloom (1999) for a discussion. Appendix A provides 
information on the MDE for all study outcomes. 

~ 5 There are four grade levels in high school (ninth to twelfth grade), so by the end of ninth grade students 
should have earned a quarter of the credits they need to graduate. 
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Table 2.4 


Outcomes and Characteristics of the Last 
Baseline Cohort of Ninth-Grade Students 


Outcome or Characteristic 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P- Value 

Ninth-grade outcomes 

Progress toward graduation 

Credits earned in core subjects 

(as % needed to graduate) 

22.15 

21.78 

0.37 

0.744 

Academic achievement 

Proficient on ELA state test (%) 

35.87 

36.23 

-0.36 

0.930 

Proficient on math state test (%) 

58.96 

58.57 

0.39 

0.922 

GPA in core subject areas (out of 4.0) 

2.15 

2.17 

-0.02 

0.822 

Behavioral outcomes (%) 

Attendance rate 

93.29 

93.19 

0.10 

0.919 

Received an in-school suspension 

21.33 

23.52 

-2.19 

0.667 

Received an out-of-school suspension 

14.32 

11.76 

2.56 

0.318 

Were expelled 

0.03 

0.02 

0.01 

0.560 

Eighth-grade outcomes and characteristics (%) 

Eligible for free/reduced-price lunch 

44.98 

47.25 

-2.26 

0.515 

English language learners 

5.42 

6.05 

-0.63 

0.530 

Proficient on 8th-grade ELA state test 

43.39 

43.82 

-0.43 

0.860 

Proficient on 8th-grade math state test 

56.43 

56.37 

0.06 

0.986 


Number of schools (total = 43) 27 16 


SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts; GPA = grade point average. 

A ninth-grade cohort is defined as a group of students who enter a high school in the same 
school year. The findings in this table are for the cohort of students who entered the study 
schools immediately prior to the implementation of NGAs. Students' ninth-grade outcomes are 
measured in students' first year of high school. Students' eighth-grade outcomes and 
characteristics are for the school year before they entered high school. 

The values in the columns labeled "NGA Schools" and "Comparison Schools" are observed 
means. Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison 
schools. The statistical significance of estimated differences is indicated as: *** = 1 percent; ** 
= 5 percent; * = 10 percent. 
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Table 2.5 


School Characteristics in the School Year Before 
NGA Implementation 


School Characteristic 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P-Value 

Eligible for free/reduced-price lunch 





(% of all students) 

31.6 

34.1 

-2.5 

0.352 

Racial/ethnic composition (% of all students) 





Black 

35.9 

33.5 

2.4 

0.580 

Hispanic 

14.7 

14.6 

0.0 

0.986 

White 

45.6 

48.1 

-2.4 

0.542 

Other 

3.8 

3.8 

0.0 

0.976 

Ninth-grade enrollment 

747.0 

674.3 

72 7 *** 

< 0.001 

Pupil-teacher ratio (all grades) 

20.1 

19.6 

0.5 

0.392 

Title 1 status (% of schools) 

3.7 

11.1 

-7.4 

0.326 

Location (% of schools) 





City 

96.3 

96.3 

0.0 

1.000 

Rural 

3.7 

3.7 

0.0 

1.000 

Number of schools (total = 43) 

27 

16 




SOURCE: Common Core of Data. 


NOTES: The values in the columns labeled "NGA Schools" and "Comparison Schools" are the 
observed means. Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison schools. 

The statistical significance of estimated differences is indicated as: *** = 1 percent; ** = 5 percent; 

* = 10 percent. 

language learners in eighth grade, and 45 percent to 47 percent of students were eligible for free 
or reduced-price lunch. 26 


26 ln Table 2.4, all information is based on the last baseline cohort of ninth-grade students. The eighth- 
grade characteristics and outcomes of students in the three follow-up cohorts were also examined, and the pat- 
tern of results looks similar (see Appendix A for these results). Appendix A also includes standard deviations 
for student outcomes that can be used to convert the baseline differences in these tables into effect sizes. To 
meet What Works Clearinghouse (2014) standards for baseline equivalence, baseline differences for student 
outcomes and characteristics must be less than 0.25 (as an effect size). All baseline student outcomes and char- 
acteristics meet this criterion. 
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Whereas Table 2.4 focuses on the outcomes of ninth-grade students, Table 2.5 turns to 
the characteristics of all students in the study schools, as well as the schools’ structural charac- 
teristics. This table shows that, in the school year before the NGAs were launched, the two 
groups of schools had similar proportions of high school students eligible for free or reduced- 
price lunch (about a third) and a similar racial and ethnic composition. The one exception is that 
the NGA schools in the study enroll a larger number of ninth-grade students than the compari- 
son schools (an average of 747 students compared with 674 students), and this difference of 73 
students is statistically significant. Substantively, this means that the NGA schools have about 
two to three additional classrooms of ninth-grade students. This difference could make it more 
challenging for the NGA schools to create a personalized learning environment and potentially 
lead to an underestimate of the effect of NGAs. That said, because ninth-grade enrollment is 
high in both groups of schools, a difference of 73 students is proportionally quite small, and 
therefore any such bias is likely to be minimal if it exists. 27 

The fact that the NGA schools and the comparison schools are indeed very similar in- 
creases the likelihood that the comparison schools’ deviations from their baseline trends provide 
the right counterfactual for what would have happened to the NGA schools had they not im- 
plemented the academies. This, in turn, strengthens this study’s ability to evaluate whether the 
NGAs caused any changes in ninth-grade student outcomes. 28 


~ 7 The pattern of similarities (and differences) in Table 2.5 also holds for the school years after the NGAs 
were created. 

28 The two groups of schools also have similar baseline trends in ninth-grade student outcomes. The size 
and the direction of the slope in these baseline trends are similar for NGA schools and comparison schools, and 
the difference in baseline trends is not statistically significant for the primary outcome (core credits earned). 
See Appendix A for further information. 
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Chapter 3 


Structural Features and Support Services 
in the Study Schools 


This chapter examines the extent to which the Ninth Grade Academies (NGAs) in this study 
were able to put in place the components of the NGA model. As discussed earlier, some NGA 
components are not unique to the NGA model, so this study also examines whether the struc- 
tures and supports of the NGA model were being used in some of the comparison schools as 
well. 1 The key findings discussed in this chapter are as follows: 

• Many of the NGAs in the study were unable to implement all four struc- 
tural components. Three years after their creation, only half the NGAs in 
the study had implemented all four components of the NGA model. 

• Some of the features of the NGA model were also part of the ninth-grade 
experience in the comparison schools. The comparison high schools in this 
study, for example, also provided various support services to their ninth- 
grade students, and a substantial percentage of them had a faculty dedicated 
to the ninth grade. 

• The main difference between the NGA schools and the comparison 
schools is with respect to space, leadership, and teaching teams. The 

NGA schools in the study were much more likely than the comparison 
schools to report having a dedicated space for ninth-grade students, a dedi- 
cated administrator watching over the ninth grade, and teaching teams. 

Overall, the NGA schools and the comparison schools do appear to differ structurally. 
However, it was not possible to measure how well or strongly the components of the NGA 
model were implemented in the study schools — only whether the components were present or 
absent. Therefore, it is unclear whether the structural differences between the NGA schools and 
the comparison schools were sufficient to have created a more personalized learning environ- 
ment for ninth-grade students in the NGA schools. 


'The findings reported in this chapter were obtained by comparing the percentages of NGA schools and 
comparison schools that implemented a particular type of structural component or support. Cluster-robust 
standard errors were used to account for the fact that some comparison schools are included in the data more 
than once. 
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Core Structural Components 

As described in Chapter 2, NGAs have four core structural features: a dedicated space, a dedi- 
cated ninth-grade administrator, a dedicated faculty, and teaching teams. By definition, all the 
NGA schools in the study implemented at least two of these four core structural components, 
and at least one of these components was a dedicated administrator or a dedicated space. (This 
was the necessary or minimum condition for a high school to be considered as having an NGA 
in this study.) Ideally, however, NGAs should include all four of these components, in order to 
maximize the conditions that will promote a more personalized learning environment for ninth - 
grade students. Accordingly, Table 3.1 also looks at the percentage of study schools that im- 
plemented each of the four core structural features during the first three years of their NGAs and 
the percentage of schools that implemented all four components. 

As shown in this table, the NGA schools in the study are more likely than the compari- 
son schools to have implemented a dedicated administrator and space. By the third year of im- 
plementation, 82 percent of the NGA schools had a dedicated space and administrator, whereas 
none of the comparison schools did. 

The NGA schools were also more likely than the comparison schools to have a dedi- 
cated ninth-grade faculty and to use teaching teams. However, the contrast between the NGA 
schools and the comparison schools is smaller for these two components, because some of the 
comparison schools were also using these strategies: 76 percent of the comparison schools 
had dedicated ninth-grade faculty by the third follow-up year, and 28 percent had teaching 
teams. This reflects the fact that team teaching and dedicated faculty are approaches that can 
be used by schools independent of implementing an NGA with its own space and its own 
administrator. 2 

Overall, the NGA schools and the comparison schools do seem to differ structurally in 
many respects. 3 However, a key question is whether the NGA schools implemented these struc- 
tural components sufficiently well to have created a more personalized experience for their 
ninth-grade students relative to the comparison schools. For reasons explained in Chapter 2, it 


2 Some of the NGA schools and comparison schools already had dedicated faculty and teaching teams be- 
fore the NGAs were created, which provides further evidence that these components can be implemented inde- 
pendent of NGAs. Appendix A provides further information on the percentage of study schools implementing 
the core components in the last baseline year. 

3 The study team also examined the contrast between the NGA schools and the comparison schools with 
respect to other structural components that are sometimes part of an NGA: a dedicated ninth-grade counselor, a 
dedicated ninth-grade social worker, and student teams. By the third year of implementation, 74 percent of the 
NGA schools in the study reported having a dedicated ninth-grade counselor, whereas only about 1 1 percent of 
comparison schools did, and 41 percent of NGA schools were teaming students, while only 18 percent of com- 
parison schools were using this strategy. Very few schools in either group had a dedicated social worker. 
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Table 3.1 


Core Structural Components of NGA Schools in Place 
in the Follow-Up Period 


Structural Component (%) 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P- Value 

Schools with each core component 

Dedicated space 

Year 1 

59.3 

0.0 

59 3 H=H=H= 

< 0.001 

Year 2 

81.5 

0.0 

81.5 *** 

< 0.001 

Year 3 

81.5 

0.0 

81.5 *** 

< 0.001 

Dedicated administrator 

Year 1 

59.3 

0.0 

59 3 *** 

< 0.001 

Year 2 

77.8 

0.0 

77 g *** 

< 0.001 

Year 3 

81.5 

0.0 

81.5 *** 

< 0.001 

Dedicated faculty 

Year 1 

81.5 

66.5 

15.0 

0.215 

Year 2 

96.3 

71.3 

25.0 ** 

0.027 

Year 3 

96.3 

76.3 

20.0 * 

0.053 

Teaching teams 

Year 1 

60.7 

20.7 

40.0 *** 

0.006 

Year 2 

79.9 

23.9 

56.0 *** 

< 0.001 

Year 3 

79.7 

27.7 

52.0 *** 

0.001 

Schools with all four core components 

Year 1 

33.3 

0.0 

33.3 *** 

< 0.001 

Year 2 

51.9 

0.0 

51 9 *** 

< 0.001 

Year 3 

51.9 

0.0 

51 9 *** 

< 0.001 

Number of schools (total = 43) 

27 

16 




SOURCE: Administrator Survey. 

NOTES: The "follow-up period" is defined as the first three years of implementation in schools that 
implemented NGAs. The values in the columns labeled "NGA Schools" and "Comparison Schools" 
are the observed means. Rounding may cause slight discrepancies in calculating sums and 
differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison schools. 
The statistical significance of estimated differences is indicated as: *** = 1 percent; ** = 5 percent; 
* = 10 percent. 
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was not possible to measure how well the components were implemented in the study schools. 
Yet it may be telling that many of the NGA schools in the study were not able to implement all 
four core structural components; only half the NGAs (14 of 27) had all these elements in place 
by the third year of implementation. 4 This suggests that finding the resources simply to put in 
place the four basic structural features of an NGA may be challenging for schools. Implement- 
ing these components well could be even harder. 


Curricular and Student Support Services 

As described in Chapter 1, NGAs often incorporate curricular and instructional support for 
ninth-grade students, as well as direct support services. Table 3.2 shows the percentage of study 
schools that offered such kinds of support at any point during the follow-up period. 5 

As shown in this table, many of the NGA schools in this study provided curricular and 
student support services to their ninth-graders. The most common types of curricular support 
were a curriculum for closing grade gaps (82 percent of NGAs), a career planning curriculum 
(74 percent), opportunities to catch up during the school day (60 percent), a college awareness 
curriculum (56 percent), and a study/social skills curriculum (56 percent). The most common 
types of student support services were extracurricular help in core subjects (85 percent of 
NGAs) and the recognition of positive behavior and achievement (70 percent). 

However, a notable proportion of the comparison schools also offered many of these 
types of support, which suggests that these services may already be part of the “business as usu- 
al” ninth-grade experience. This is perhaps to be expected, given that these kinds of support do 
not require an NGA structure. Given their broader prevalence, the contrast between the NGA 
schools and the comparison schools with respect to these support services is relatively small. 
One area where the NGA schools and their comparisons differ is with respect to a career plan- 
ning curriculum: 74 percent of NGA schools implemented a career planning curriculum, where- 
as only 16 percent of comparison schools did so. The NGA schools were also more likely to 
have a college preparation curriculum (30 percent of NGA schools compared with none of the 
comparison schools), but this difference predates the creation of NGAs, so it cannot be attribut- 
ed to the their implementation. 6 


4 This does not appear to be driven by any one component: By the third year of implementation, each com- 
ponent was in place in at least 80 percent of schools. 

5 The results are very similar when broken down by follow-up year. 

6 See Appendix A for further information about the percentage of NGA schools and comparison schools 
that offered the curricular and instructional support services in the last baseline year. 
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Table 3.2 


Curricular and Student Support Services for Ninth-Grade Students 

in the Follow-Up Period 


Type of Support (%) 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P-Value 

Curricular and instructional support 

Career planning curriculum 

74.1 

16.4 

57 7 H=H=H= 

<0.001 

College awareness curriculum 

55.6 

44.0 

11.5 

0.536 

College preparation curriculum 

29.6 

0.0 

29.6 *** 

0.005 

Curriculum for closing grade gaps 

81.5 

77.6 

3.8 

0.762 

Opportunities for catch-up during school day 

59.3 

48.1 

11.1 

0.417 

Curriculum for study/social skills 

55.6 

63.0 

-7.4 

0.609 

Flexible scheduling for student regrouping 

18.5 

18.5 

0.0 

1.000 

Student support 

Summer bridge or orientation program 

44.4 

66.7 

-22.2 

0.182 

Extracurricular help in core subjects 

85.2 

77.8 

7.4 

0.497 

Data-driven response to academics and behavior 

51.9 

59.3 

-7.4 

0.597 

Recognition of positive behavior/achievement 

70.4 

51.9 

18.5 

0.232 

Number of schools (total = 43) 

27 

16 




SOURCE: Administrator Survey. 


NOTES: The "follow-up period" is defined as the first three years of implementation in schools that 
implemented NGAs. The values in the columns labeled "NGA Schools" and "Comparison Schools" are 
the observed means. Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison schools. The 
statistical significance of estimated differences is indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 


Another pattern that is worth noting (not shown in Table 3.2) is that the percentage of 
study schools offering these support services increased over time. This is true in both the NGA 
schools and the comparison schools. For example, an additional 48 percent of comparison 
schools used a curriculum to close grade gaps in the follow-up period (relative to the last base- 
line year). 7 These increases could have been driven by some statewide policy change in Florida 
that prompted schools to ramp up their services, or by a greater general awareness of the im- 
portance of supporting ninth-grade students. 


7 See Appendix A for further information about the percentage of schools that offered curricular and in- 
structional support in the last baseline year. 
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Summary 

An important finding to emerge from this chapter is that some of the features of the NGA 
model are also part of the ninth-grade experience in the comparison schools. The comparison 
high schools in this study, for example, provided various kinds of support services to their 
ninth-grade students, and many of them have a faculty dedicated to ninth grade. This practical 
reality confirms that what distinguishes the NGA model is not its individual components, but 
rather the idea that when the components are put in place together, they should interact to 
promote an environment in which adults make a concerted effort to be aware of and respon- 
sive to student needs. Unfortunately, it is not possible to measure whether the NGA schools in 
this study implemented the components of the model in a way that would be characterized as 
strong or ideal implementation, and in turn, whether this produced the more personalized 
learning environment for ninth-grade students intended by the NGA theory of action. Howev- 
er, this study can look at whether the NGA schools were able to achieve their ultimate goal: 
improving ninth-grade student outcomes. 
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Chapter 4 


Effects on Student Progress, 
Academic Achievement, and Behavior 


This chapter looks at whether the structural differences between the schools implementing 
Ninth Grade Academies (NGAs) and the comparison schools in this study were sufficient to 
produce improvements in ninth-grade students’ progress toward graduation, their academic 
achievement, and their behavioral outcomes. The effect of NGAs is examined for the first three 
cohorts of students to enroll in the study schools, corresponding to the first three years of NGA 
implementation. The key findings discussed in this chapter are as follows: 

• Overall, there is no conclusive evidence that the NGAs in this study im- 
proved ninth-grade students’ outcomes. After the NGAs were created, 
ninth-grade students in the NGA schools actually performed worse than pre- 
dicted (relative to their schools’ baseline trends) on a range of outcomes. 
However, so did students at the comparison schools. This suggests that some 
policy change may have happened in the state of Florida that caused a gen- 
eral decline in student outcomes, and that creating NGAs did not prevent the 
study schools from experiencing this decline. 

• Nor is there any conclusive evidence that the subgroup of NGA schools 
that implemented all four core structural components had an effect on 
ninth-grade students’ outcomes. This suggests that the NGA schools — 
even those that implemented all of the core NGA components — may not 
have been able to implement these components as well as intended. 

As explained in Chapter 2, the effect of the NGAs in this study is the difference be- 
tween the average deviations from baseline trend for the NGA schools and the comparison 
schools. This is different from a randomized experiment, where the impact of an intervention is 
obtained by comparing the mean outcomes of the program group and the control group in the 
follow-up period. Therefore, the tables in this chapter include information on deviations from 
baseline trend, as opposed to mean outcomes. These deviations can be useful for helping to in- 
terpret the impact findings: They provide contextual in formation on whether ninth-grade stu- 
dents in the follow-up cohorts performed better or worse than expected relative to earlier co- 
horts of students. Box 4.1 explains how to interpret the tables in this chapter. 1 The statistical 
significance of estimated effects is assessed up to the 10 percent level in this study. 


'Appendix A includes information on the statistical model used to produce the results. 
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Box 4.1 


How to Read the Tables in This Chapter 

The tables in this chapter show the estimated effect of NGAs on student outcomes by follow- 
up cohort (Cohorts 1, 2, and 3, representing the first three years of NGA implementation). 
A ninth-grade cohort is defined as a group of students who enter high school in the same 
school year. 

The columns in the tables present the following information: 

• Deviation from baseline trend. The first two columns of results show the estimated devi- 
ation from their baseline trend for NGA schools and the matched comparison schools for 
each outcome by follow-up cohort. For all outcomes except disciplinary infractions, posi- 
tive deviations from trend indicate that ninth-grade students in the NGA schools or the 
comparison schools did better than expected in the follow-up period given their baseline 
trends, while negative deviations from trend indicate that students did worse than ex- 
pected. For disciplinary outcomes (suspensions and expulsions), the reverse is true. The 
statistical significance of estimated deviations from trend is indicated for levels of 1 per- 
cent (ttt), 5 percent (ff), and 10 percent (f). 

• Estimated effect. The third column of results is the estimated effect of the NGAs for 
each outcome by follow-up cohort. The estimated effect is the difference between the 
NGA schools’ deviation and the comparison schools’ deviation from their baseline trends 
(column 1 minus column 2). 

• Confidence interval. The next two columns in the table are the lower and upper bounds 
of the 90 percent confidence interval for the estimated effects of the NGAs. Generally 
speaking, the true impact of NGAs has a 90 percent probability of falling somewhere 
within this interval. 

• P-value for estimated effect. The last column of the table shows the p-value for each 
estimated effect. The p-value indicates the probability of finding the estimated effect (or 
a larger effect) if there were in fact no difference between the NGA schools and the 
comparison schools with respect to their deviations from their trends. In this study, an 
estimated effect that has a p-value of 1 0 percent or less is considered “statistically signif- 
icant” because it is unlikely that this estimated effect would be observed if the NGA 
schools and comparison schools did not have different deviations from trend. The num- 
ber of asterisks indicates whether the estimated effect is statistically significant at the 1 
percent (***), 5 percent (**), or 10 percent (*) level. 

The rows in the tables represent the outcome and follow-up cohort (Cohort 1, Cohort 2, and 
Cohort 3) associated with each estimated deviation from baseline trend and each estimated 
effect. 
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Effects on Ninth-Grade Student Outcomes 


Figure 4. 1 starts by examining the trends in credits earned in core subject areas in ninth grade 
(as a percentage of core credits required for graduation), which is the primary outcome in this 
study. In the years before the NGAs were created, the percentage of credits earned by students 
was on the rise in both the NGA schools and the comparison schools. However, after the NGAs 
were launched, deviations from baseline trend for the NGA schools were negative, which indi- 
cates that ninth-grade students in the follow-up cohorts earned fewer credits than would have 
been predicted based on earlier cohorts’ credit earning. Ninth-grade students in the comparison 
schools also performed worse than expected, by a similar amount. 2 As Table 4.1 indicates, the 
estimated effect of NGAs on this outcome is not statistically significant for the first three co- 
horts of ninth-grade students to enroll in them, and it cannot be concluded that NGAs helped 
students earn more credits in ninth grade. 

Nor do the NGAs in this study appear to have improved students’ academic achieve- 
ment (Table 4.1). Similar to the pattern of results for credits earned, ninth-grade students in the 
follow-up period were less likely to be proficient on state tests than predicted by the baseline 
trend for this outcome, and this is true for both the NGA schools and the comparison schools. 
Thus, the NGAs do not show a positive effect on students’ state test scores. In fact, the estimat- 
ed effect on math proficiency is statistically negative for the first follow-up cohort, because the 
comparison schools’ deviations from trend are not as negative as those of the NGA schools. 
However, this result should be interpreted cautiously, because these negative effects do not 
stand up in some of the sensitivity analyses that were conducted. 3 (The nature of these analyses 
are discussed later in this chapter.) 

With respect to students’ grade point average (GPA) in core subjects, at first glance the 
findings seem slightly more encouraging. The estimated effect of the NGAs on this outcome, 
though not statistically significant, is positive. However, closer inspection of the results reveals 
that the average GPA of students did not actually improve in the NGA schools. The positive 
direction of the effect is driven by the fact that students’ average GPA was lower than predicted 
for both groups of schools, but more so in the comparison schools than in the NGA schools. 

Finally, as Table 4.2 shows, the NGAs in this study do not appear to have improved 
students’ behavioral outcomes during the first three years of implementation. Mirroring the 
pattern for credits earned and proficiency on state tests, ninth-grade students in the follow-up 


2 As a secondary analysis, the effect of NGAs on core credits attempted (as a percentage of the credits 
needed for graduation) was also examined. The pattern of results for this outcome is very similar to the pattern 
for credits earned (estimated effects are negative in direction and not statistically significant). This suggests that 
the trends in Figure 4.1 are being driven by factors affecting credit taking. 

3 See Appendix A for these findings. 
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Figure 4.1 


Baseline Trends and Deviations from Baseline Trend in the Percentage of 
Core Credits Earned in Ninth Grade 



Ninth-grade cohort 


SOURCE: Student records from the Florida Department of Education. 

NOTE: The estimated effect of NGA schools is equal to the estimated deviation from baseline trend for the NGA schools minus the estimated deviation from 
baseline trend for the comparison schools. The estimated effect of NGA schools shown in this figure is not statistically significant at the 10 percent level. 



Table 4.1 


Estimated Effect of NGAs on School Progress and Academic Achievement in Ninth Grade 



Deviation from Baseline Trend 


90% Confidence Interval 

P- Value for 

Outcome 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Effect 

Lower 

Bound 

Upper 

Bound 

Estimated 

Effect 

Progress toward graduation 

Credits earned in core subjects 
(as % needed to graduate) 

Follow-Up Cohort 1 

-1.98 ttt 

-1.56 ff 

-0.42 

-1.93 

1.09 

0.646 

Follow-Up Cohort 2 

-2.36 fff 

-2.20 ff 

-0.16 

-2.19 

1.87 

0.897 

Follow-Up Cohort 3 

-3.24 fff 

-1.92 

-1.31 

-3.87 

1.24 

0.397 

Academic achievement 

Proficient on ELA state test (%) 

Follow-Up Cohort 1 

0.39 

0.71 

-0.32 

-2.53 

1.89 

0.813 

Follow-Up Cohort 2 

2.23 ff 

1.43 

0.80 

-2.08 

3.67 

0.647 

Follow-Up Cohort 3 

2.47 ff 

0.82 

1.65 

-1.64 

4.94 

0.409 

Proficient on math state test (%) 

Follow-Up Cohort 1 

-2.90 fff 

-0.22 

-2.68 * 

-5.09 

-0.26 

0.068 

Follow-Up Cohort 2 

-3.49 fff 

-0.79 

-2.70 

-5.59 

0.20 

0.125 

Follow-Up Cohort 3 

-3.44 ff 

-1.29 

-2.15 

-5.63 

1.32 

0.307 

GPA in core subject areas (out of 4.0) 

Follow-Up Cohort 1 

-0.01 

-0.06 f 

0.05 

-0.03 

0.13 

0.309 

Follow-Up Cohort 2 

-0.01 

-0.08 ff 

0.07 

-0.02 

0.16 

0.177 

Follow-Up Cohort 3 

-0.06 

-0.09 f 

0.03 

-0.07 

0.14 

0.619 

Number of schools (total = 43) 

27 

16 
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Table 4.1 (continued) 


SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts; GPA = grade point average. 

A ninth-grade cohort is defined as a group of students who enter a high school in the same school 
year. The follow-up cohorts are the first three cohorts of students who entered the study schools after 
the NGAs were launched. Ninth-grade outcomes are measured in students' first year of high school. 

The values in the column labeled "NGA Schools" are the estimated deviations from baseline trend 
for schools that implemented an NGA. The "Comparison Schools" values are the estimated deviations 
from baseline trend for matched comparison schools. The values in the "Estimated Effect" column are 
the differences between NGA schools and comparison schools with respect to their deviations from 
baseline trend. Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to estimated deviations and estimated differences between NGA 
schools and comparison schools. The statistical significance of estimated deviations is indicated as: 
fff = 1 percent; f f = 5 percent; f = 10 percent. The statistical significance of estimated effects is 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


cohorts generally had lower attendance rates and were more likely to be suspended than pre- 
dicted by the baseline trends for these outcomes. This is true for both the NGA schools and 
the comparison schools, so the estimated effect of NGAs on these outcomes is not statistically 
significant (and is generally in the wrong direction). 


Exploratory Analyses 

Several additional analyses were conducted to explore various hypotheses about when and for 
whom the NGAs in this study might have been effective. 

First, the effect of the NGAs on students’ outcomes in their second year of high school 
(tenth grade) was examined. The underlying hypothesis is that ninth-grade students may have 
benefited from NGAs in ways that were not captured by the outcomes in this study, and that it 
takes time for these benefits to manifest themselves as effects on students’ academic and behav- 
ioral outcomes. Alternatively, because students cannot be identified as at risk for dropping out 
of high school until they have completed some of their course work and have failed classes, ad- 
ditional support for these students may not be triggered until later in ninth grade, in which case 
effects on student outcomes may not emerge until tenth grade. Neither of these hypotheses is 
supported by the data, however: The NGAs in this study do not appear to have had an effect on 
students’ tenth-grade outcomes, so there is no evidence of any delayed effects. 

Second, the effect of NGAs was examined for subgroups of students who have higher 
academic needs, including English language learners and students who were not proficient on 
their eighth-grade reading and math state tests. The underlying hypothesis is that NGAs are per- 
haps most beneficial to students who are struggling academically and most at risk of dropping 
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Table 4.2 


Estimated Effect of NGAs on Behavioral Outcomes in Ninth Grade 



Deviation from Baseline Trend 


90% Confidence Interval 

P- Value for 


NGA 

Comparison 

Estimated 

Lower 

Upper 

Estimated 

Outcome (%) 

Schools 

Schools 

Effect 

Bound 

Bound 

Effect 

Attendance rate 







Follow-Up Cohort 1 

-1.26 ff 

-0.73 

-0.54 

-1.92 

0.84 

0.523 

Follow-Up Cohort 2 

-1.82 ftf 

-1.11 

-0.71 

-2.42 

1.00 

0.494 

Follow-Up Cohort 3 

-2.14 ftf 

-1.76 f 

-0.38 

-2.35 

1.58 

0.748 

Received an in-school suspension 







Follow-Up Cohort 1 

0.77 

-0.37 

1.15 

-3.75 

6.05 

0.699 

Follow-Up Cohort 2 

0.10 

0.56 

-0.47 

-6.03 

5.09 

0.890 

Follow-Up Cohort 3 

-1.17 

2.04 

-3.21 

-10.17 

3.76 

0.448 

Received an out-of-school suspension 







Follow-Up Cohort 1 

0.46 

0.73 

-0.27 

-3.36 

2.81 

0.884 

Follow-Up Cohort 2 

2.13 

3.47 fff 

-1.34 

-4.97 

2.28 

0.542 

Follow-Up Cohort 3 

2.49 

4.62 fff 

-2.13 

-6.52 

2.25 

0.423 

Were expelled 







Follow-Up Cohort 1 

-0.01 

0.01 

-0.02 

-0.07 

0.04 

0.570 

Follow-Up Cohort 2 

-0.01 

0.04 

-0.05 

-0.11 

0.02 

0.241 

Follow-Up Cohort 3 

-0.02 

0.05 

-0.06 

-0.14 

0.01 

0.155 

Number of schools (total = 43) 

27 

16 





(continued) 
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Table 4.2 (continued) 


SOURCE: Student records from the Florida Department of Education. 

NOTES: A ninth-grade cohort is defined as a group of students who enter a high school in the same 
school year. The follow-up cohorts are the first three cohorts of students who entered the study schools 
after the NGAs were launched. Ninth-grade outcomes are measured in students' first year of high 
school. 

The values in the column labeled "NGA Schools" are the estimated deviations from baseline trend 
for schools that implemented an NGA. The "Comparison Schools" values in the next column are the 
estimated deviations from baseline trend for matched comparison schools. The values in the "Estimated 
Effect" column are the differences between NGA schools and comparison schools with respect to their 
deviations from baseline trend. Rounding may cause slight discrepancies in calculating sums and 
differences. 

A two-tailed t-test was applied to estimated deviations and estimated differences between NGA 
schools and comparison schools. The statistical significance of estimated deviations is indicated as: 
fff = 1 percent; f f = 5 percent; f = 10 percent. The statistical significance of estimated effects is 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


out. Yet the NGAs in this study do not appear to have improved the outcomes of lower- 
performing students. 

Finally, the effect of NGAs was separately examined for the 14 NGA schools that im- 
plemented all four core structural components during the follow-up period (dedicated space, 
dedicated administrator, dedicated faculty, and teaching teams). The hypothesis is that these 
schools, because they implemented all core structural components, were able to set up an organ- 
izational structure that was more conducive to creating a personalized learning environment for 
ninth-grade students. Once again, the results did not support the hypothesis; estimated effects on 
student outcomes for these 14 NGA schools are similar to the results for the full sample. One 
possible reason for this pattern of results is that the implementation of the four core components 
in these schools may have been weak, and that the intended improvements in personalization 
that characterize a strong version of the NGA model may not have occurred. 


Sensitivity Analyses 

Several sensitivity analyses were conducted to explore whether the comparison schools in this 
study provided the right counterfactual for the NGA schools and the extent to which the results 
from this study are rigorous. 4 

• Changes in school and student characteristics. The NGA schools and 
comparison schools in this study may have experienced different demograph- 
ic shifts in the characteristics of their ninth-grade cohorts during the follow- 

4 See Appendix A for the findings from these analyses. 
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up period. If this were true, then the comparison schools would not provide 
the right counterfactual for the NGA schools . 5 To investigate this threat to 
validity, the effect of NGAs on student outcomes was reestimated, control- 
ling for the average characteristics of students in each cohort and the struc- 
tural characteristics of schools . 6 The findings from this analysis are similar to 
those presented earlier in this chapter, which suggests that the results from 
this study are not influenced by demographic changes. 

• Propensity score matching. When choosing the comparison schools for this 
study, the “similarity” between NGA schools and potential comparison 
schools was measured using the Euclidean distance metric, which captures 
the distance between two schools across several matching variables . 7 To test 
whether the results of this study were sensitive to the matching method, 
comparison schools were rechosen using propensity score matching instead 
(another method of measuring the “similarity” between schools). The impact 
of NGAs was reestimated using the comparison schools selected through this 
alternative method. The estimated effects from this sensitivity analysis are 
similar to those from the main analysis. 

• Design validation analysis. One of the advantages of having access to 
several years of baseline data is that the results of the comparative inter- 
rupted time series (CITS) design can be validated by looking at the estimat- 
ed “effect” of NGAs on students in the last baseline cohort. By definition, 
this effect should be zero, because the NGAs were not yet being imple- 
mented; thus this provides a useful benchmark for validating the design and 
the selection of comparison schools. This sensitivity analysis was conduct- 
ed by pretending that ninth-grade students who were enrolled in the school 
year previous to NGA implementation (the last baseline cohort) were the 
first follow-up cohort in the CITS design. The selection of comparison 
schools was then conducted again using the newly defined baseline period 
(now excluding the last baseline cohort), and the effects of NGAs on stu- 
dent outcomes were reestimated. This analysis found that the “effects” of 


5 This can be thought of as selection bias. See Shadish, Cook, and Campbell (2002). 

Specifically, the analysis controls for school characteristics (percentage of students by racial/ethnic group, 
ninth-grade enrollment, pupil-teacher ratio) in the year that each cohort entered the study schools, as well as the 
eighth-grade characteristics of each cohort of students (percentage of English language learners, percentage 
eligible for free or reduced-price lunch, percentage proficient on the English language arts and math state tests). 

7 The Euclidean distance across M matching variables is defined as D — jY.M(T m — C m ) 2 , where T m is 
the value of characteristic m for the NGA school and C m is the value of characteristic m for the non-NGA 
school. 
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NGAs on the outcomes of the last baseline cohort are close to zero in mag- 
nitude and not statistically significant, and that estimated effects for subse- 
quent cohorts are similar to those from the main analysis. 

Taken together, the findings from these sensitivity analyses suggest that the comparison 
schools are similar enough to the NGA schools to provide a convincing counterfactual, which in 
turn lends greater credibility to the findings presented in this report. 


Summary and Discussion 

The findings in this chapter provide no conclusive evidence that the NGAs in the study im- 
proved ninth-grade students’ outcomes. In fact, both the NGA schools and the comparison 
schools performed worse than predicted on a range of outcomes. This decline in student out- 
comes is particularly striking because, as discussed in Chapter 3, support services actually in- 
creased during this time in both groups of schools. One hypothesis for this pattern of results is 
that the study schools were trying to do too much at the same time, which reduced the quality of 
the support services provided and ultimately student outcomes. Another hypothesis is that ex- 
ternal factors umneasured in this study (such as a new statewide accountability system) precipi- 
tated a decline in all schools’ outcomes during the years when most of the study schools created 
their NGAs. If this is true, then the findings from this study suggest that creating NGAs did not 
help schools stem this decline. Even NGA schools that implemented all four core structural 
components do not appear to have improved student outcomes. This finding suggests that the 
study schools may not have been able to implement the core NGA components well enough to 
create a more personalized learning environment for ninth-grade students. 
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Chapter 5 

Discussion 


The findings from this study suggest that creating effective Ninth Grade Academies (NGAs) 
can be difficult. The NGAs in this study do not appear to have improved ninth-grade students’ 
progress toward graduation, their academic achievement, or their behavioral outcomes. Even 
among the subgroup of NGA schools that implemented all four core structural components, 
student outcomes do not seem to have improved. One hypothesis for this pattern of results is 
that the core NGA components may not have been implemented as intended in the study 
schools, and that as a result, the staff was not able to create a more personalized ninth-grade ex- 
perience for students. 

This hypothesis is supported by the findings of an earlier study conducted by MDRC 
that examined the implementation of NGAs in Broward County, one of the school districts in 
this study. 1 Several high schools in Broward County launched NGAs starting in the 2005-2006 
school year, when the district received an $8 million, five-year grant from the federal Smaller 
Learning Communities (SLC) program. Then, in fall 2007, the Broward school district led an 
effort to support NGAs, which prompted another wave of NGA creation in the district. 
MDRC’s study examined the implementation of NGAs in 18 Broward high schools — to leam 
about the context of the initiative, the extent to which NGAs were implemented as intended, and 
the conditions associated with strong and weak implementation. 2 MDRC’s study found that of 
the 1 8 high schools in the district that tried to implement the core structural features of NGAs, 
only 3 were able to establish strong NGAs. 3 A faculty dedicated to the ninth grade and interdis- 


Lcgtcrs. Parise, and Rappaport (2013). 

2 The implementation study conducted by MDRC in Broward included 18 NGAs. Of these NGAs, 10 are 
included in the present study. The remaining 8 are not included because they created their NGAs too early or 
too late to be used in the comparative interrupted time series analysis. As a subgroup analysis, the effect of the 
10 Broward NGAs on student outcomes was separately estimated. There is no evidence that these NGAs im- 
proved ninth-grade student outcomes. 

3 These high schools were rated as having “strong” implementation of the first three NGA components 
(dedicated space, administrator, and faculty) and at least “moderate” implementation of the interdisciplinary 
teaching teams component during the first three years of the NGA. Schools with strong implementation of the 
dedicated space component had a self-contained space where all ninth-graders spend the majority of their day, 
and this space housed the classrooms of virtually all teachers teaching ninth-grade classes, had offices and ad- 
ministrative support staff, and had a visible identity as an NGA. Schools with strong implementation of the 
dedicated administrator component had an administrator who was knowledgeable about the NGA, engaged 
almost exclusively with ninth-grade students and faculty, and was committed to making the NGA his or her top 
priority. Schools with strong implementation of the dedicated faculty component were those where all ninth- 
grade teachers taught exclusively ninth-graders, identified themselves as ninth-grade teachers as opposed to 
subject-area teachers, and expressed belief in the NGA concept. Schools with moderate implementation of the 
interdisciplinary teaching team component were ones where teacher teams were at least present in some way. 
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ciplinary teaching teams were the most challenging components for schools to implement at a 
level that was considered strong . 4 

Broward County provides an interesting case study for understanding the challenges of 
creating strong NGAs without external assistance. In Broward, one of the main barriers to 
strong NGA implementation was a lack of financial and technical support to schools from the 
district. Most schools did not have the necessary in-house expertise to implement the features of 
NGAs on their own. Creating strong interdisciplinary teaching teams — and getting ninth-grade 
teachers from different content areas to work together — proved to be especially difficult. Yet 
schools did not receive any additional funds to train ninth-grade teachers on how to work more 
effectively with each other across disciplines to meet the specific needs of ninth-grade students. 
As a result, though most administrators and teachers in Broward understood what an NGA 
should look like conceptually, they did not seem to know how to enact NGAs on a daily basis. 

The implementation of NGAs in Broward was further undermined by the way in which 
the district chose to define NGAs. The district emphasized the structural aspects of the acade- 
mies (dedicated space, a dedicated administrator, a dedicated faculty, and interdisciplinary 
teaching teams), rather than how school staff members should work within these structures. Ac- 
cordingly, few NGA administrators and principals in Broward considered what it would take to 
use these structures to improve teacher collaboration and to create a more personalized envi- 
ronment for students. 

Finally, the strength of Broward’s NGAs was compromised by competing district prior- 
ities. In addition to promoting NGAs, the district turned its attention to data-driven progress 
monitoring and subject-area professional learning communities. While not inconsistent with 
NGAs, these efforts were implemented independently of NGAs, so they competed for scarce 
district resources. 

These three challenges — to varying degrees — are likely to have played out in all or 
most of the NGA schools in the present study, for two reasons. First, over a third of the schools 
in the study sample are located in Broward County. Second, the non-Broward NGAs in this 
study do not appear to have been created by a district-wide policy (they were probably created 
by individual high schools), so these NGAs may have had access to even less support than the 
NGAs in Broward. 

Thus Broward’s experience suggests three key lessons for schools and districts that 
have decided to use the NGA model. First, beyond implementing the structural features of 


4 Strong implementation of the dedicated faculty component is defined in the previous footnote. Strong 
implementation of the teaching team component is defined as follows: Almost all ninth-grade students were 
taught by teachers in interdisciplinary teams, and teachers strongly identified as being part of their teams and 
met regularly to identify and address student needs collectively. 
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NGAs, plans to create NGAs should build in processes and fonns of support to engage staff and 
help them take advantage of the NGA structure to personalize ninth-grade students’ experience. 
Second, schools need specific guidelines, on-site support, training for teachers, secure resources, 
and tools to guide practice if they are to implement strong and self-sustaining NGAs that suc- 
ceed in personalizing the learning environment. Finally, plans to create NGAs should be coor- 
dinated with other district initiatives intended to improve student outcomes so that NGAs are 
not weakened by competing priorities. 

More recently, Broward County has been piloting three types of staff-related support to 
strengthen and intensify its NGAs: (1) a teacher leader who develops and implements academic 
and social support services for ninth-graders; (2) data-response teams that meet weekly to re- 
view student data, identify and monitor struggling students, and respond with appropriate aca- 
demic and behavioral interventions; and (3) formal partnerships with organizations to support 
tutoring, mentoring, and other enriching activities for ninth-graders. The vehicle for promoting 
these enhancements is a Community of Practice (CoP) that includes a team from each partici- 
pating school. The CoP meets regularly to develop action plans for implementing these new 
practices, to receive additional training, and to reflect on progress and share challenges. MDRC 
has been studying these enhancements. 5 

The idea of pairing NGAs with intensive monitoring and other support has also made 
its way into existing high school reform models that include NGAs. Talent Development Sec- 
ondary, for example, has recently partnered with two other organizations — City Year and 
Communities In Schools — to create the Diplomas Now high school reform model. 6 Talent 
Development Secondary’s role in this partnership is to implement various types of instructional 
and structural refomis, including NGAs. City Year provides near-peer volunteers (recent col- 
lege graduates) who assist teachers in the classroom, monitor attendance and behavior, and pro- 
vide coaching, tutoring, mentoring, homework support, and extended day activities to students. 
Communities In Schools rounds out the partnership by providing school-wide supports aimed 
at improving the school climate, as well as case management for the students most at risk of 
dropping out. 7 

Although combining NGAs with monitoring and other support makes sense in theory, 
whether this can improve student outcomes is an open question. Supplementing the NGA model 
with other elements increases the number of program features that schools must put in place, 
thus leaving schools with more limited time and bandwidth to create strong NGAs at the core of 
their approach. For example, MDRC is currently evaluating the implementation and impact of 
the Diplomas Now model based on a school-level randomized experiment. Of the schools in the 


5 Legters and Parise (2016). 

6 Sepanik et al. (2015). 

7 There is also a Diplomas Now middle school model, which does not include NGAs. 
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MDRC study that are implementing the Diplomas Now model, only 60 percent were able to put 
in place an NGA with adequate or high fidelity during the first two years of implementation, 
and only 29 percent had interdisciplinary teaching teams that met multiple times a week to dis- 
cuss students. 8 

Given the continuing popularity of NGAs and recent efforts to strengthen them, further 
research is needed to understand their implementation and effects. Several districts have at- 
tempted to create NGAs in the last few years. 9 The findings in this report — which are based on 
NGAs created a decade ago — may not generalize to these more recent attempts to create 
NGAs, which may have benefited from some of the lessons offered by earlier efforts such as 
Broward’s. In order to provide the most rigorous evidence of their effectiveness, it would also 
be of benefit to the field for researchers to partner with districts that would be willing to ran- 
domly select which high schools will implement future NGAs, because as of yet all studies of 
NGAs have been based on quasi-experimental research designs. These studies should also 
measure how well the NGA components are being implemented and whether they create a more 
personalized learning enviromnent, so that the NGA theory of action can be fully understood 
and tested. 


8 Sepanik et al. (2015). Interim impact findings from the study are forthcoming. 

"’Appendix B provides background information on several other school districts that have created NGAs 
more recently. 
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Appendix A 

Technical Information 




This appendix provides information on various technical aspects of the analysis. The first sec- 
tion presents response rates for the student outcomes in the analysis. The second section de- 
scribes the statistical model used to estimate the effect of Ninth Grade Academies (NGAs). The 
third section discusses the minimum detectable effect sizes for the study’s outcomes. The fourth 
section presents additional infomiation on the baseline outcomes and characteristics of students 
in this analysis. The fifth section provides the standard deviations for the measures in this study. 
The sixth section presents the results of the sensitivity analyses discussed in Chapter 4. The final 
section presents information on the percentage of schools that implemented the NGA compo- 
nents in the baseline period. 


Response Rates for Student Outcomes 

As explained in Chapter 2, the student outcomes data for this study come from student rec- 
ords provided by the Florida Department of Education. Because these data are administrative, 
they are available for almost all students enrolled in the study schools. Appendix Table A.l 
shows the percentage of students for whom outcomes data are available, for the last baseline 
cohort of students and the three follow-up cohorts. Response rates range from 87 percent to 
100 percent, depending on the outcome. For any given outcome, the response rate is similar 
for the NGA schools and the comparison schools. Although response rates do differ by a sta- 
tistically significant amount for credits earned and grade point average (GPA), the difference 
is very small in magnitude and still qualifies as “low attrition” based on What Works Clear- 
inghouse standards. 1 


Statistical Model 

In this study, the effect of NGAs is estimated using a comparative interrupted time series 
(CITS) design. Due to issues related to data processing, the student-level data used for the anal- 
ysis were aggregated by school and ninth-grade cohort for the purposes of conducting the anal- 
ysis (rather than being used at the student level). The effect of NGAs was then estimated by fit- 
ting the following model to the aggregated data set: 2 

Y jt = T.j YjSCHOOL jt + Y,} (pjSCHOOL jt * RELYE AR t + a 1 FYl t + & NGAj * 

FYl t + a 2 FY2 t + /3 2 NGAj * FY2 t + a 3 FY3 t + /3 3 NGAj * FY3 t + e Jt 


'What Works Clearinghouse (2014). 

2 The analysis does not weight each observation by the number of students in each school, based on rec- 
ommendations from the field related to using school-level data (Jacob, Goddard, and Kim, 2014). 
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Appendix Table A.l 


Response Rates for Ninth-Grade Outcomes 


Outcome (%) 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P- Value 

Last baseline cohort 

Credits earned in core subjects 

92.2 

93.5 

-1.3 

0.114 

Retained in 9th grade 

100.0 

100.0 

0.0 

1.000 

Proficient on ELA state test 

87.0 

88.4 

-1.4 

0.192 

Proficient on math state test 

86.9 

87.9 

-1.0 

0.366 

GPA in core subject areas 

92.2 

93.5 

-1.3 

0.114 

Attendance rate 

96.9 

97.6 

-0.7 

0.103 

Received an in-school suspension 

100.0 

100.0 

0.0 

1.000 

Received an out-of-school suspension 

100.0 

100.0 

0.0 

1.000 

Were expelled 

100.0 

100.0 

0.0 

1.000 

Follow-Up Cohort 1 

Credits earned in core subjects 

92.5 

93.8 

-1.3 ** 

0.016 

Retained in 9th grade 

100.0 

100.0 

0.0 

1.000 

Proficient on ELA state test 

87.4 

88.0 

-0.6 

0.480 

Proficient on math state test 

87.2 

87.7 

-0.6 

0.520 

GPA in core subject areas 

92.5 

93.8 

-1.3 ** 

0.016 

Attendance rate 

97.0 

97.4 

-0.4 

0.334 

Received an in-school suspension 

100.0 

100.0 

0.0 

1.000 

Received an out-of-school suspension 

100.0 

100.0 

0.0 

1.000 

Were expelled 

100.0 

100.0 

0.0 

1.000 

FoIIow-Ud Cohort 2 

Credits earned in core subjects 

93.1 

94.2 

-1.1 * 

0.056 

Retained in 9th grade 

100.0 

100.0 

0.0 

1.000 

Proficient on ELA state test 

87.1 

88.7 

-1.6 

0.158 

Proficient on math state test 

87.1 

88.6 

-1.5 

0.156 

GPA in core subject areas 

93.1 

94.2 

-1.1 * 

0.056 

Attendance rate 

96.7 

97.6 

-1.0 * 

0.098 

Received an in-school suspension 

100.0 

100.0 

0.0 

1.000 

Received an out-of-school suspension 

100.0 

100.0 

0.0 

1.000 

Were expelled 

100.0 

100.0 

0.0 

1.000 


(continued) 
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Appendix Table A.l (continued) 


Outcome (%) 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P- Value 

Follow-Up Cohort 3 





Credits earned in core subjects 

93.4 

94.7 

-1.3 ** 

0.045 

Retained in 9th grade 

100.0 

100.0 

0.0 

1.000 

Proficient on ELA state test 

88.5 

89.6 

-1.1 

0.246 

Proficient on math state test 

88.2 

89.4 

-1.2 

0.222 

GPA in core subject areas 

93.4 

94.7 

-1.3 ** 

0.045 

Attendance rate 

97.1 

97.2 

0.0 

0.912 

Received an in-school suspension 

100.0 

100.0 

0.0 

1.000 

Received an out-of-school suspension 

100.0 

100.0 

0.0 

1.000 

Were expelled 

100.0 

100.0 

0.0 

1.000 

Number of schools (total = 43) 

27 

16 




SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts; GPA = grade point average. 

A ninth-grade cohort is defined as a group of students who enter a high school in the same 
school year. The last baseline cohort is the cohort of students who entered the study schools 
immediately prior to the implementation of the NGAs. The follow-up cohorts are the first three 
cohorts of students who entered the study schools after the NGAs were launched. Response rates 
are for outcomes measured in students' first year of high school. 

A two-tailed t-test was applied to estimated deviations and estimated differences between 
NGA schools and comparison schools. The statistical significance of estimated differences is 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


where j denotes schools and t denotes school year (or equivalently, the ninth-grade cohort for a 
given school year). In a CITS design, school years (cohorts) are measured relative to the follow- 
up period: t is equal to 1,2, and 3 for the first three follow-up cohorts of ninth-grade students, 
and is 0 or negative for the baseline cohorts. 3 

The variables in the model are defined as follows: 

Yj t = the school-level outcome for cohort t in school j. 


3 The baseline and follow-up cohorts for a comparison school are the same as the NGA school to which it 
is matched (that is, the definitions of RELYEAR, FYl, FY1. and FY3 are the same for an NGA school and its 
matched comparison school). Also, some schools have data available for five baseline cohorts, while others 
have data for four baseline cohorts. The statistical model can accommodate this variability in the length of the 
baseline period. 
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SCHOOL j 
RELYEAR t 

NGAj 

FYl t 

FY2 t 

FY3 t 

£ jt 


a set of /dichotomous indicators for school (= 1 for school j and 0 
otherwise). 

a continuous variable for school year (cohort) centered at the last 
baseline cohort for a school, such that RELYEAR = 0 is the last base- 
line cohort. 

a dichotomous indicator for whether school j is an NGA school (= 1 if 
school is an NGA school, 0 if a matched comparison school). 

a dichotomous indicator for the first follow-up cohort to be enrolled in 
an NGA school (= 1 if yes, 0 otherwise). 

a dichotomous indicator for the second follow-up cohort to be en- 
rolled in an NGA school (= 1 if yes, 0 otherwise). 

a dichotomous indicator for the third follow-up cohort to be enrolled 
in an NGA school (= 1 if yes, 0 otherwise). 

the random variation in the outcome of interest across cohorts within 
schools (within-school variation). 


Note that in this model, each school has its own baseline intercept and slope. This strat- 
egy is used to account for the longitudinal nature of the data, whereby school years are clustered 
within schools. 


From the model, the following quantities of interest can be obtained: 


yj 


<pj 

a 1 


«! + /?! 


the predicted mean (intercept) for each NGA school and comparison 
school in the study in the last baseline year. 

the baseline slope for each school in the study. 

the deviation from baseline trend for the first follow-up cohort in the 
comparison schools. 

the deviation from baseline trend for the first follow-up cohort in the 
NGA schools. 


a 2 = the deviation from baseline trend for the second follow-up cohort in 
the comparison schools. 

a 2 + ft 2 = the deviation from baseline trend for the second follow-up cohort in 

the NGA schools. 


48 



a 3 = the deviation from baseline trend for the third follow-up cohort in the 

comparison schools. 

a 3 + /? 3 = the deviation from baseline trend for the third follow-up cohort in the 

NGA schools. 

It is important to note that in this model, /?j represents the estimated impact of NGAs in 
the first year of their implementation — that is, the deviation from baseline trend for NGA 
schools minus the deviation from trend for comparison schools. Similarly, (l 2 represents the 
estimated impact of NGAs in the second year of implementation, and /? 3 represents the estimat- 
ed impact in the third year. 

As explained in Chapter 2, some comparison schools were chosen as the “match” for 
more than one NGA school. Thus, some comparison schools are in the data set more than once. 
To account for these duplicate observations, cluster-robust standard errors are used for hypothe- 
sis testing. 


Minimum Detectable Effects 

Appendix Table A.2 shows the minimum detectable effects (MDEs) for this study. Formally, 
the MDE is the smallest true program impact that can be detected with a reasonable degree of 
power (in this case, 80 percent) for a given level of statistical significance (in this case, 10 per- 
cent for a two-tailed test). These MDEs were obtained by multiplying the standard error of the 
estimated impact by 2.5. Note that the number of schools — which is an important determinant 
of the MDE because it affects the standard error — varies across outcomes depending on how 
much historical data are available for the outcome. For state test scores, historical data are not 
available for as many school years, so the NGA schools that started earlier, and thus do not have 
data for four baseline cohorts, are excluded from the analysis. Also note that in a CITS design, 
the MDE is larger for later follow-up years (that is, larger for estimated effects in Year 3 com- 
pared with Year 1), because the reliability of trend projections decreases with the distance in 
time for which the projection is being made. 4 


Supplemental Baseline Information 

Appendix Table A.3 presents the eighth-grade characteristics and outcomes of students in the 
three follow-up cohorts. This table shows that the ninth-grade students enrolled in the NGA 


4 Bloom (1999). 
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Appendix Table A.2 


Minimum Detectable Effects for Ninth-Grade Student Outcomes, 
by Follow-Up Cohort and Sample 

Minimum Detectable Effect 


Outcome 

Number of Schools 


(10% level) 


NGA 

Schools 

Comparison 

Schools 

Follow-Up 

Cohort 1 

Follow-Up 

Cohort 2 

Follow-Up 

Cohort 3 

Credits earned in core subjects 






(as % needed to graduate) 

27 

16 

2.28 

3.08 

3.87 

Proficient on ELA state test (%) 

25 

14 

3.35 

4.36 

4.99 

Proficient on math state test (%) 

25 

14 

3.67 

4.40 

5.28 

GPA in core subject areas (out of 4.0) 

27 

16 

0.13 

0.14 

0.16 

Attendance rate (%) 

27 

16 

2.09 

2.60 

2.98 

Received an in-school suspension (%) 

27 

16 

7.43 

8.43 

10.56 

Received an out-of-school suspension (%) 

27 

16 

4.67 

5.50 

6.65 

Were expelled (%) 

27 

16 

0.08 

0.10 

0.11 


SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts; GPA = grade point average. 

A ninth-grade cohort is defined as a group of students who enter a high school in the same school year. The 
follow-up cohorts are the first three cohorts of students who entered the study schools after the NGAs were 
launched. Ninth-grade outcomes are measured in students' first year of high school. The minimum detectable 
effects in this table are calculated by multiplying the standard error of the estimated effects by 2.5. A statistical 
significance level of 10 percent is assumed. 


schools and in the comparison schools had similar baseline characteristics and academic per- 
formance as they entered high school. 5 

Appendix Tables A.4 and A.5 show the estimated baseline intercept and baseline slope, 
respectively, from the impact model described earlier in this appendix. In the impact model, 
each school has its own baseline intercept and slope. Therefore, to estimate the average inter- 
cept and slope across all NGA schools and comparison schools, the school-level slopes ( <pj ) and 
intercepts (Yj ) from the regression model are weighted by the precision of the estimated slope 
for each school. 6 


5 As an effect size, all baseline differences are less than 0.25, which is the criterion used by the What 
Works Clearinghouse (2014) to determine baseline equivalence. 

6 The precision of a school-level slope is the squared inverse of the standard error of the slope. 
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Appendix Table A.3 


Eighth-Grade Characteristics and Outcomes of Ninth-Grade 
Students in the Follow-Up Cohorts 


Characteristic or Outcome in 8th Grade (%) 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P- Value 

Follow-Up Cohort 1 

Eligible for free/reduced-price lunch 

47.3 

49.1 

-1.9 

0.570 

English language learners 

6.2 

6.1 

0.1 

0.900 

Proficient on 8th-grade ELA state test 

43.4 

43.8 

-0.4 

0.882 

Proficient on 8th-grade math state test 

57.9 

59.2 

-1.3 

0.626 

Follow-Up Cohort 2 

Eligible for free/reduced-price lunch 

46.9 

48.9 

-2.0 

0.570 

English language learners 

6.0 

6.6 

-0.6 

0.608 

Proficient on 8th-grade ELA state test 

45.3 

44.8 

0.5 

0.867 

Proficient on 8th-grade math state test 

60.1 

59.9 

0.1 

0.953 

Follow-Up Cohort 3 

Eligible for free/reduced-price lunch 

47.9 

50.1 

-2.3 

0.528 

English language learners 

6.1 

6.2 

0.0 

0.986 

Proficient on 8th-grade ELA state test 

47.6 

46.5 

1.2 

0.723 

Proficient on 8th-grade math state test 

62.3 

60.0 

2.4 

0.452 

Number of schools (total = 43) 

27 

16 




SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts. 

A ninth-grade cohort is defined as a group of students who enter a high school in the 
same school year. The follow-up cohorts are the first three cohorts of students who entered 
the study schools after the NGAs were launched. Students' eighth-grade outcomes and 
charateristics are for the school year before they entered high school. 

The values in the columns labeled "NGA Schools" and "Comparison Schools" are the 
observed means. Rounding may cause slight discrepancies in calculating sums and 
differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison 
schools. The statistical significance of estimated differences is indicated as: *** = 1 
percent; ** = 5 percent; * = 10 percent. 
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Appendix Table A.4 


Intercept of the Baseline Trend in Ninth-Grade Student Outcomes 


Outcome 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P-Value 

Progress toward graduation 

Credits earned in core subjects 
(as % needed to graduate) 

22.74 

22.50 

0.24 

0.197 

Academic achievement 

Proficient on ELA state test (%) 

36.04 

33.70 

2 34 H=H=H= 

<0.001 

Proficient on math state test (%) 

62.42 

57.80 

4.62 *** 

<0.001 

GPA in core subject areas (out of 4.0) 

2.16 

2.23 

-0.08 *** 

<0.001 

Behavioral outcomes (%) 

Attendance rate 

94.04 

93.14 

0 9Q *** 

<0.001 

Received an in-school suspension 

20.61 

18.33 

2 27 *** 

<0.001 

Received an out-of-school suspension 

11.66 

11.87 

-0.21 

0.558 

Were expelled 

0.01 

-0.01 

0.02 *** 

<0.001 

Number of schools (total = 43) 

27 

16 




SOURCE: Student records from the Florida Department of Education. 


NOTES: ELA = English language arts; GPA = grade point average. 

The values in this table are the intercept of the estimated baseline trend in outcomes based on 
the four or five cohorts of students who entered the study schools prior to the implementation of 
NGAs. A ninth-grade cohort is defined as a group of students who enter a high school in the 
same school year. Ninth-grade outcomes are measured in students' first year of high school. 

Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison schools. 
The statistical significance of estimated differences is indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 


Appendix Table A.4 presents the estimated baseline intercept for the NGA schools and 
the comparison schools. These intercepts represent the predicted mean outcomes for NGA 
schools and comparison schools in the last baseline year (in Figure 2.1, this is the predicted val- 
ue of the outcome for Cohort 0). The difference between the predicted means (intercepts) of the 
NGA schools and the comparison schools is larger than the difference in their observed baseline 
means (Table 2.4). However, as an effect size, none of the differences in predicted means are 
greater than 0.25, which is the criterion used by the What Works Clearinghouse (2014) for base- 
line equivalence. 
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Appendix Table A.5 


Slope of the Baseline Trend in Ninth-Grade Student Outcomes 


Outcome 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P-Value 

Progress toward graduation 

Credits earned in core subjects 
(as % needed to graduate) 

0.50 

0.43 

0.07 

0.307 

Academic achievement 

Proficient on ELA state test (%) 

2.13 

1.26 

0 gy *** 

<0.001 

Proficient on math state test (%) 

3.08 

2.00 

1.08 *** 

<0.001 

GPA in core subject areas (out of 4.0) 

0.01 

0.03 

-0.02 *** 

<0.001 

Behavioral outcomes (%) 

Attendance rate 

0.43 

0.45 

-0.02 

0.634 

Received an in-school suspension 

0.21 

0.35 

-0.14 

0.514 

Received an out-of-school suspension 

-0.98 

-0.97 

-0.01 

0.959 

Were expelled 

0.00 

-0.01 

0.01 *** 

<0.001 

Number of schools (total = 43) 

27 

16 




SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts; GPA = grade point average. 

The values in this table are the estimated baseline slope in outcomes based on the four or five 
cohorts of students who entered the study schools prior to the implementation of NGAs. A ninth- 
grade cohort is defined as a group of students who enter a high school in the same school year. 
Ninth-grade outcomes are measured in students' first year of high school. 

Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison schools. 
The statistical significance of estimated differences is indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 


Appendix Table A.5 presents the slope of the baseline trends of the NGA schools and 
the comparison schools. The values in this table represent the average annual change in each 
outcome during the baseline period, or the slope of the baseline trends. Positive slopes indicate 
that the outcome’s value was increasing during the baseline period, while negative slopes indi- 
cate that the outcome’s value was decreasing. With the exception of expulsions, the direction of 
the slopes for all outcomes is the same for the NGA schools and the comparison schools (both 
are positive, or both are negative). 
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Standard Deviations 


Appendix Tables A.6, A.7, and A.8 provide standard deviations for the baseline characteristics 
and outcomes discussed in the study for the schools and students in the study. Appendix Table 
A.6 presents the standard deviations for students’ eighth-grade outcomes for the last baseline 
cohort and the first three follow-up cohorts. Appendix Table A.7 presents the standard devia- 
tions for the last baseline cohort’s ninth-grade outcomes. Appendix Table A.8 presents the 
standard deviations for school characteristics from the Common Core of Data in the last base- 
line school year before the NGAs were created. Note that the standard deviations for student 
characteristics and outcomes (Appendix Tables A.6 and A.7) are student-level standard devia- 
tions, which is preferable for making effect sizes comparable across students. 7 For school char- 
acteristics from the Common Core of Data (Appendix Table A.8), standard deviations are at the 
school level by definition. Standard deviations are shown for the NGA schools and comparison 
schools separately, and for both groups of schools pooled together. 


Sensitivity Analyses 

As explained in Chapter 4, several sensitivity analyses were conducted to explore whether the 
comparison schools provide the right counterfactual and whether the findings are rigorous. The 
results of these analyses are presented in Appendix Table A.9 (for academic outcomes) and Ap- 
pendix Table A. 10 (for behavioral outcomes). 

For there to be convincing evidence of positive effects, the estimated effect of the 
NGAs on a particular outcome should be in the right direction across all the sensitivity analyses 
and for all follow-up cohorts, as well as statistically significant. In addition, the estimated effect 
for the last baseline cohort of students (in the “design validation” analysis) should be small and 
not statistically significant, because this cohort was enrolled before the NGAs were imple- 
mented. 


Overall, the sensitivity analyses find no evidence that the NGAs in the study improved 
student outcomes. Although there is a statistically significant reduction in expulsions for Co- 
horts 2 and 3 in the “design validation” analysis, the estimated effect of NGAs on this outcome 
for the last baseline cohort of students is also statistically significant, which is not plausible. 
This suggests that the estimated effects on expulsions for Cohorts 2 and 3 in the “design valida- 
tion” analysis are not credible. Similarly, although the NGAs have a statistically negative effect 
on students’ perfomiance on math state tests in the main analysis and in one of the sensitivity 
analyses, this result does not hold in the other two sensitivity analyses. 


7 What Works Clearinghouse (2014). 
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Appendix Table A.6 


Standard Deviation of the Eighth-Grade Characteristics and 
Outcomes of Ninth-Grade Students in the Study Cohorts 


Characteristic or Outcome in 8th Grade (%) 

NGA 

Schools 

Comparison 

Schools 

All 

Schools 

Last Baseline Cohort 

Eligible for free/reduced-price lunch 

49.6 

49.9 

49.8 

English language learners 

23.2 

24.3 

23.7 

Proficient on 8th-grade ELA state test 

49.7 

49.5 

49.6 

Proficient on 8th-grade math state test 

49.4 

49.7 

49.6 

Follow-Up Cohort 1 

Eligible for free/reduced-price lunch 

49.9 

50.0 

50.0 

English language learners 

24.5 

24.5 

24.5 

Proficient on 8th-grade ELA state test 

49.6 

49.5 

49.6 

Proficient on 8th-grade math state test 

49.3 

49.3 

49.3 

Follow-Up Cohort 2 

Eligible for free/reduced-price lunch 

49.9 

50.0 

50.0 

English language learners 

24.5 

25.3 

24.9 

Proficient on 8th-grade ELA state test 

49.8 

49.6 

49.8 

Proficient on 8th-grade math state test 

48.9 

49.1 

49.0 

Follow-Up Cohort 3 

Eligible for free/reduced-price lunch 

49.9 

50.0 

50.0 

English language learners 

25.4 

24.3 

24.9 

Proficient on 8th-grade ELA state test 

50.0 

49.9 

49.9 

Proficient on 8th-grade math state test 

48.2 

49.0 

48.6 

Number of schools 

27 

16 

43 


SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts. 

A ninth-grade cohort is defined as a group of students who enter a high 
school in the same school year. The last baseline cohort is the cohort of students 
who entered the study schools immediately prior to the implementation of the 
NGA. The follow-up cohorts are the first three cohorts of students who entered 
the study schools after the NGA was launched. Students' eighth-grade outcomes 
and characteristics are for the school year before they entered high school. 

The values in this table are the student-level standard deviation of the 
outcome or characteristic for each group of schools. 
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Appendix Table A.7 

Standard Deviation of the Ninth-Grade Outcomes of the 
Last Baseline Cohort of Ninth-Grade Students 


School Characteristic 

NGA 

Schools 

Comparison 

Schools 

All 

Schools 

Progress toward graduation 

Credits earned in core subjects 
(as % needed to graduate) 

10.25 

9.98 

10.12 

Academic achievement 

Proficient on ELA state test (%) 

47.83 

47.73 

47.78 

Proficient on math state test (%) 

48.95 

49.38 

49.16 

GPA in core subject areas (out of 4.0) 

1.01 

1.01 

1.01 

Behavioral outcomes (%) 

Attendance rate 

10.14 

9.80 

9.98 

Received an in-school suspension 

40.46 

41.42 

40.93 

Received an out-of-school suspension 

34.21 

31.88 

33.14 

Were expelled 

1.67 

1.11 

1.43 

Number of schools 

27 

16 

43 


SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts; GPA = grade point average. 

A ninth-grade cohort is defined as a group of students who enter a high school 
in the same school year. The last baseline cohort is the cohort of students who 
entered the study schools immediately prior to the implementation of the NGA. 
The follow-up cohorts are the first three cohorts of students who entered the 
study schools after the NGA was launched. Students' ninth-grade outcomes are 
measured in students' first year of high school. 

The values in this table are the student-level standard deviation of the outcome 
or characteristic for each group of schools. 
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Appendix Table A.8 


Standard Deviation of School Characteristics in the 
Last Baseline Year 


School Characteristic 

NGA 

Schools 

Comparison 

Schools 

All 

Schools 

Eligible for free/reduced-price lunch 

(% of all students) 

13.9 

12.5 

13.2 

Racial-ethnic composition (% of all students) 

Black 

29.2 

17.8 

24.0 

Hispanic 

11.7 

7.7 

9.8 

White 

26.3 

20.1 

23.2 

Other 

2.4 

1.9 

2.1 

Ninth-grade enrollment 

173.8 

153.9 

166.7 

Pupil-teacher ratio (all grades) 

1.5 

2.9 

2.3 

Title 1 status (% of schools) 

19.2 

32.0 

26.4 

Location (% of schools) 

City 

19.2 

19.2 

19.1 

Rural 

19.2 

19.2 

19.1 

Number of schools 

27 

16 

43 


SOURCE: Common Core of Data. 


NOTES: The last baseline year is defined as the school year immediately prior to the 
implementation ofNGAs. 

The values in this table are the school-level standard deviation of the 
characteristic for each group of schools. 
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Appendix Table A.9 


Estimated Effect of NGAs on School Progress and Academic Achievement in 
Ninth Grade, by Sensitivity Analysis 





Adjusted 

Propensity 


Main 

Design 

for School 

Score 

Outcome 

Sample 

Validation' 

Characteristics 11 

Matching 1 " 

Progress toward graduation 

Credits earned in core subjects 
(as % needed to graduate) 





Last baseline cohort 


-0.08 



Follow-Up Cohort 1 

-0.42 

-0.84 

-0.55 

-0.78 

Follow-Up Cohort 2 

-0.16 

-0.98 

-0.27 

-0.50 

Follow-Up Cohort 3 

-1.31 

-1.84 

-1.10 

-0.94 

Academic achievement 





Proficient on ELA state test (%) 





Last baseline cohort 


-0.25 



Follow-Up Cohort 1 

-0.32 

-0.51 

0.42 

-1.67 

Follow-Up Cohort 2 

0.80 

-0.65 

1.05 

0.03 

Follow-Up Cohort 3 

1.65 

0.30 

1.85 

-1.35 

Proficient on math state test (%) 





Last baseline cohort 


-1.16 



Follow-Up Cohort 1 

-2.68 * 

-3.02 

-2.50 * 

-1.31 

Follow-Up Cohort 2 

-2.70 

-3.33 

-2.76 * 

0.53 

Follow-Up Cohort 3 

-2.15 

-3.13 

-2.40 

-1.89 

GPA in core subject areas (out of 4.0) 





Last baseline cohort 


-0.04 



Follow-Up Cohort 1 

0.05 

-0.05 

0.05 

0.03 

Follow-Up Cohort 2 

0.07 

-0.05 

0.07 

0.01 

Follow-Up Cohort 3 

0.03 

-0.14 * 

0.05 

-0.04 

Number of schools 

43 

41 

43 

43 


(continued) 
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Appendix Table A.9 (continued) 


SOURCE: Student records from the Florida Department of Education. 

NOTES: ELA = English language arts. 

A ninth-grade cohort is defined as a group of students who enter a high school in the same school year. 
The last baseline cohort is the cohort of students who entered the study schools immediately prior to the 
implementation of the NGAs. The follow-up cohorts are the first three cohorts of students who entered 
the study schools after the NGAs were launched. Ninth-grade outcomes are measured in students' first 
year of high school. 

The values in the "Estimated Effect" column are the differences between NGA schools and 
comparison schools with respect to their deviations from baseline trend. A two-tailed t-test was applied to 
estimated differences between NGA schools and comparison schools. The statistical significance of 
estimated effects is indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a The selection of comparison schools was conducted using the same process as for the main study 
sample, but in this case pretending that NGA schools’ last baseline cohort was the first cohort to be 
affected by the NGA. If the study design and matching approach are sound, then the “effect” for the last 
baseline cohort should be close to zero in magnitude and not statistically significant, and impacts in the 
follow-up years should be similar to impacts for the main sample. 

b The impact estimates are adjusted for the characteristics of schools (percentages of students by 
racial/ethnic group, ninth-grade enrollment, pupil-teacher ratio) in the school year when each cohort 
entered the study schools, as well as the eighth-grade characteristics of each cohort of students 
(percentage of English language learners, percentage with an individualized education plan, percentage 
eligible for free or reduced-price lunch, percentage proficient on the ELA and math state tests). 

c Comparison schools were chosen using propensity scores as an overall measure of the difference 
between schools, rather than the Euclidean distance. 
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Appendix Table A.10 


Estimated Effect of NGAs on Behavioral Outcomes in 
Ninth Grade, by Sensitivity Analysis 



Main 

Design 

Adjusted 
for School 

Propensity 

Score 

Outcome (%) 

Sample 

Validation 3 

Characteristics 11 

Matching 1- 

Attendance rate 





Last baseline cohort 


-0.09 



Follow-Up Cohort 1 

-0.54 

-0.95 

-0.64 

-1.21 

Follow-Up Cohort 2 

-0.71 

-1.26 

-0.69 

-1.52 

Follow-Up Cohort 3 

-0.38 

-0.96 

-0.34 

-1.82 

Received an in-school suspension 





Last baseline cohort 


-4.29 



Follow-Up Cohort 1 

1.15 

-2.88 

1.35 

0.75 

Follow-Up Cohort 2 

-0.47 

-5.97 

-0.53 

-2.04 

Follow-Up Cohort 3 

-3.21 

-7.18 

-3.61 

-2.47 

Received an out-of-school suspension 





Last baseline cohort 


-2.89 



Follow-Up Cohort 1 

-0.27 

-1.96 

-0.23 

2.23 

Follow-Up Cohort 2 

-1.34 

-3.89 

-1.33 

1.97 

Follow-Up Cohort 3 

-2.13 

-4.26 

-2.02 

2.61 

Were expelled 





Last baseline cohort 


-0.06 * 



Follow-Up Cohort 1 

-0.02 

-0.05 

-0.02 

0.01 

Follow-Up Cohort 2 

-0.05 

-0.09 ** 

-0.05 

-0.01 

Follow-Up Cohort 3 

-0.06 

-0.09 ** 

-0.06 

-0.02 

Number of schools 

43 

41 

43 

43 


(continued) 
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Appendix Table A.10 (continued) 


SOURCE: Student records from the Florida Department of Education. 

NOTES: A ninth-grade cohort is defined as a group of students who enter a high school in the same school 
year. The last baseline cohort is the cohort of students who entered the study schools immediately prior to the 
implementation of the NGAs. The follow-up cohorts are the first three cohorts of students who entered the 
study schools after the NGAs were launched. Ninth-grade outcomes are measured in students' first year of 
high school. 

The values in the "Estimated Effect" column are the difference between NGA schools and comparison 
schools with respect to their deviations from baseline trend. A two-tailed t-test was applied to estimated 
deviations and estimated differences between NGA schools and comparison schools. The statistical 
significance of estimated effects is indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a The selection of comparison schools was conducted using the same process as for the main study sample, 
but in this case pretending that NGA schools’ last baseline cohort was the first cohort to be affected by the 
NGA. If the study design and matching approach are sound, then the “effect” for the last baseline cohort 
should be close to zero in magnitude and not statistically significant, and impacts in the follow-up years 
should be similar to impacts for the main sample. 

b The impact estimates are adjusted for the characteristics of schools (percentages of students by 
racial/ethnic group, ninth-grade enrollment, pupil-teacher ratio) in the school year when each cohort entered 
the study schools, as well as the eighth-grade characteristics of each cohort of students (percentage of English 
language learners, percentage with an individualized education plan, percentage eligible for free or reduced- 
price lunch, percentage proficient on the ELA and math state tests). 

c Comparison schools were chosen using propensity scores as an overall measure of the difference between 
schools, rather than the Euclidean distance. 


NGA Components and Support Services in the Last 
Baseline Year 

Appendix Tables A.l 1 and A. 12 look at the percentages of schools that were implementing the 
NGA structural components and support services in the school year before the NGAs were cre- 
ated. As shown in Appendix Table A. 1 1, as one would expect, none of the study schools had a 
dedicated administrator or space the year before the NGAs were created. Some of the NGA 
schools and comparison schools did have a dedicated faculty and teaching teams; this is also to 
be expected, because these strategies can be used outside of NGAs. For similar reasons, some of 
the NGA schools and comparison schools were also offering different types of support services 
in the last baseline year (Appendix Table A. 12). However, as noted in Chapter 3, the percentage 
of schools offering these services was greater in the follow-up period (Table 3.2) than in the 
baseline period (Appendix Table A. 12) among both the NGA schools and the comparison 
schools. 
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Appendix Table A. 11 

Core Structural Components of NGAs in Place in the Last Baseline Year 


Structural Component (%) 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P- Value 

Dedicated space 

0.0 

0.0 

0.0 

1.000 

Dedicated administrator 

0.0 

0.0 

0.0 

1.000 

Dedicated faculty 

29.6 

64.6 

-35.0 ** 

0.039 

Teaching teams 

3.3 

15.3 

-12.0 

0.192 

Schools with all four core components 

0.0 

0.0 

0.0 

1.000 

Number of schools (total = 43) 

27 

16 




SOURCE: Administrator Survey. 

NOTES: The "last baseline year" is defined as the school year before the NGAs were created. 
The values in the columns labeled "NGA Schools" and "Comparison Schools" are the observed 
means. Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison schools. 
The statistical significance of estimated differences is indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 
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Appendix Table A. 12 


Curricular and Student Support Services for Ninth-Grade Students 

in the Last Baseline Year 


Type of Support (%) 

NGA 

Schools 

Comparison 

Schools 

Estimated 

Difference 

P- Value 

Curricular and instructional suDDort 

Career planning curriculum 

14.8 

11.0 

3.8 

0.714 

College awareness curriculum 

11.1 

11.1 

0.0 

1.000 

College preparation curriculum 

22.2 

0.0 

22.2 *** 

0.007 

Curriculum for closing grade gaps 

37.0 

29.3 

7.7 

0.539 

Opportunities for catch-up during school day 

25.9 

29.6 

-3.7 

0.745 

Curriculum for study/ social skills 

29.6 

18.1 

11.5 

0.345 

Flexible scheduling for student regrouping 

11.1 

18.8 

-7.7 

0.497 

Student support 

Summer bridge or orientation program 

7.4 

7.4 

0.0 

1.000 

Extracurricular help in core subjects 

48.1 

32.8 

15.4 

0.250 

Data-driven response to academics and behavior 

22.2 

45.3 

-23.1 

0.153 

Recognition of positive behavior/achievement 

40.7 

13.8 

26.9 ** 

0.038 

Number of schools (total = 43) 

27 

16 




SOURCE: Administrator Survey. 


NOTES: The "last baseline year" is defined as the school year before the NGAs were created. The 
values in the columns labeled "NGA Schools" and "Comparison Schools" are the observed means. 
Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between NGA schools and comparison schools. The 
statistical significance of estimated differences is indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 
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Appendix B 


Beyond the Sunshine State: Ninth Grade Academies 
in Other School Districts 




MDRC sought to go beyond its evaluation of Ninth Grade Academies (NGAs) in Florida. This 
appendix reports the results of a supplementary study, a qualitative investigation into efforts to 
create NGAs in nine other school districts, shown in Appendix Table B.l, located across the 
country. 1 Eight of the nine school districts were included because they were awarded funding 
through the 2010 Smaller Learning Communities (SLC) grant competition sponsored by the 
U.S. Department of Education. A review of the abstracts of the proposals that the districts sub- 
mitted to the department indicated that Ninth Grade Academies (also referred to as Freshman 
Academies, or FAs) were a prominent feature of the SLC high school refomis that were 
planned for 16 high schools across these eight districts. 2 The ninth district is the District of 
Columbia, which undertook a major new initiative to implement NGAs in its nine comprehen- 
sive high schools beginning in the 2013-2014 school year. 

In March and April 2016, MDRC researchers conducted interviews ranging in length 
from 45 to 70 minutes with personnel in all nine districts to leam about their efforts in response 
to the grants; a total of 14 interviews were completed. The interviews revealed that all but one 
district that proposed to implement an NGA as part of its 2010 SLC grant actually did so. 3 The 
experiences of the eight districts that did, and of the 23 schools discussed in district interviews, 
provide a useful counterpoint to those of the Florida schools and districts that are the focus of 
the larger report. 4 

In brief, the interviews confirm the findings of the larger study that implementing all 
four of the components of an NGA that were described in the body of the report is challeng- 
ing, and that consistent support and oversight by individuals in positions of authority is need- 
ed to ensure maximum effectiveness. Turnover among administrators and competing priori- 
ties have interfered with and sometimes upended academy functioning. At the same time, 
however, the interview results suggest that, even without all four components, NGAs can 


For convenience, the term “school districts’’ in this appendix includes the Hawaii State Department of 
Education and the West Orange (NJ) Board of Education. “District X” in the table requested anonymity. 

‘All 2010 SLC grantees whose proposal abstracts clearly pointed to plans to implement NGAs are includ- 
ed in the study. 

3 West Orange Board of Education, the SLC grantee that that did not implement the proposed NGA, did 
put in place other support services for freshmen, including a summer “bridge” program for rising ninth-graders 
with low academic skills, a half-year “freshman seminar” course incorporating organizational and study skills, 
and mentoring by students in the upper grades. In addition, one District of Columbia high school that began to 
implement an NGA did not do so to the satisfaction of district administrators, and the special funding for the 
NGA that the school received was eliminated after one year. 

4 The 23-school figure excludes West Orange High School, where an NGA was never implemented. The 
Hawaii State Department of Education’s SLC grant involved five high schools; the study obtained information 
about four of these. 
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Appendix Table B.l 


Characteristics of NGAs in the Supplementary Study 


School District or Agency and 

Participating Schools 

Year NGA 
Began 

Still in Existence 
as of Spring 2016 

Dedicated 

Space 

Components 

Dedicated Teaching 

Administrator T eams a 

Dedicated 

Faculty* 5 

Fairbanks North Star Borough 

School District (AK) 

Lathrop High School 

2006 

No 

0 

X 

x c 

X 

Woodland Joint Unified School District (CA) 

Woodland High School 

2007 

No 

X d 

X 

X 

X 

Pioneer High School 

2011 

No 

0 

X 

0 

X 

Hawaii State Department of Education 

Castle High School 

2011 

Yes 

0 

X 

X 

X 

Kapa'a High School 

2011 

Yes 

X 

X 

X 

X 

Maui High School 

2005 

Yes 

0 

X 

X 

X 

McKinley High School 

2005 

Yes 

0 

X 

X 

X 

Beverly Public Schools (MA) 

2011 

Yes 

x d 

X 

X 

X 

Springfield School District 19 (OR) 

Springfield High School 

2011 

Yes 

0 

X 

0 

X 

Thurston High School 

2011 

Yes 

0 

X 

x c 

X 

Bradley County/Cleveland City Schools (TN) 

Bradley Central High School 

2008 

Yes 

X 

X 

X 

X 

Walker Valley High School 

2008 

Yes 

X 

X 

X 

X 

Cleveland High School 

2007 

No 

X 

X 

X 

X 


(continued) 



Appendix Table B.l (continued) 


School District or Agency and 

Participating Schools 

Year NGA 
Began 

Still in Existence 
as of Spring 2016 

Dedicated 

Space 

Components 

Dedicated Teaching 

Administrator T eams a 

Dedicated 

Faculty 13 

District X 

Unnamed high school 

2011 

Yes 

X 

X 

X 

X 

District of Columbia Public Schools (DC) 

Anacostia High School 

2013 

Yes 

X 

X 

X 

X 

Ballou High School 

2013 

Yes 

X 

X 

X 

X 

Cardozo High School 

2013 

Yes 

X 

X 

X 

X 

Coolidge High School 

2013 

Yes 

X 

X 

X 

X 

Dunbar High School 

2013 

Yes 

X 

X 

X 

X 

Eastern High School 

2013 

Yes 

X 

X 

X 

X 

Roosevelt High School 

2013 

Yes 

X 

X 

X 

X 

Wilson High School 

2013 

No 

X 

X 

X 

X 

Woodson High School 

2013 

Yes 

X 

X 

X 

X 


SOURCE: Interviews with school district or agency personnel. 

NOTES: X = School is implementing the component. All districts except the District of Columbia were awarded funding from the U.S. 
Department of Education’s 2010 Smaller Learning Communities grant program. 

a An NGA is considered to have a teaching team if the team consists of teachers in the four core subjects (English, mathematics, 
social studies, and science), with exceptions as noted. 

b Dedicated faculty is defined as teachers who teach ninth-graders primarily or exclusively. 

c The NGA does not include math teachers. 

d Science labs are not located within the NGA’s dedicated space. 



create more personalized environments for students. 5 A number of interview respondents, too, 
noted that student outcomes at their schools had improved after NGAs were put in place. 
While it cannot be concluded that the NGAs produced these better outcomes — rigorous re- 
search methods would be needed to establish such a causal connection — they give reason to 
believe that the impact story contained in this report may not be the final or definitive one. 
Further evaluation of the NGA approach would seem to be in order. 

The next section of this appendix describes the characteristics of the schools included in 
this scan and of the NGAs that were implemented in these settings. The final section summariz- 
es key points and lessons emerging from the interviews. 


Characteristics of Schools and NGAs Included in the Scan 

The 23 high schools with NGAs varied in a number of ways. They were located in large ur- 
ban centers and suburban and rural settings and ranged in size from 433 to 1,980 students. 
Over half (13 of the 23, or 56 percent) were schools serving large proportions of low-income 
students and receiving federal funding under Title I. Ninth-grade retention rates ranged from 
zero to 38 percent. 6 

Appendix Table B.l shows that the schools receiving SLC grants began to implement 
their NGAs by 2011. The interviewers learned that several schools had begun implementing 
NGA-like structures before receiving the 2010 grants; in these cases, the grants were generally 
used to expand the NGAs to include all ninth-graders (not just some of them) and to support 
professional development for NGA teachers and administrators. 

Only 16 of the 23 schools’ NGAs included a separate physical space, an academy ad- 
ministrator, and teachers in the four core subject areas (English, mathematics, social studies, and 
science) who taught ninth-graders primarily or exclusively. Some NGAs appeared to be func- 
tioning effectively in the absence of one or another of these elements; the section that follows 
elaborates on this point. 

The NGAs differed in whether they included students repeating ninth grade. For exam- 
ple, all the NGAs in the District of Columbia excluded repeaters as a matter of policy, while at a 
high school in another district where the number of repeaters is low, students were reassigned to 
the NGA but with a different group of teachers from the preceding year. 


5 For one thing, some NGAs have been subdivided into smaller substructures (sometimes called “houses” 
or “teams”) comprising 90 to 100 students who take their classes together and share the same group of core- 
subject teachers. 

6 Data on school size and Title I status come from the Common Core of Data for the 2013-2014 school 
year. Statistics on ninth-grade retention rates are based on U.S. Office of Civil Rights data for the 2011-2012 
school year. 
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At the time of the interviews, 11 of the 15 NGAs in the study that were supported in 
part through the SLC grants, along with 8 of the 9 original NGAs in the District of Columbia, 
continued to operate. The following section considers the factors that led NGAs to be sustained 
over time or, conversely, that led to their demise. 


Principal Findings from the Interviews 

• Some elements of the program model appear to be more important than 
others to a well-functioning NGA: While a separate space in which all 
ninth-grade classrooms are located may be helpful to forging a separate 
sense of ninth-grade identity, it does not appear to be essential, and some 
teams have functioned well without a faculty dedicated to the ninth 
grade in all four core subject areas. Some schools could not locate all the 
ninth-grade core subject classrooms together. A couple of respondents, for 
example, noted that while other ninth-grade classes could be housed together, 
science labs, with their special equipment, could not readily be moved to a 
different place in the building. In another district, the former project director 
for an NGA effort that appears to be operating successfully in two schools 
remarked that moving the NGA teachers into a separate space would have 
“upset the apple cart.” The NGAs in that district do not include math teach- 
ers, since ninth-graders are enrolled in many different math classes. 

• Common planning time for teachers is essential, according to interview 
respondents; in the NGAs that were studied, this time has primarily 
been devoted to discussions of individual students and how to help them, 
but it has also been used to consider ways to make instruction deeper 
and more engaging. NGA teachers typically met during their common 
planning time once or twice a week. In these meetings, they identified stu- 
dents who were struggling and talked about how to assist them in dealing 
with both academic and socioemotional issues through extra attention, in- 
school and after-school tutoring, and coordination of other services. Conver- 
sation also centered on strengthening instruction, planning field trips and in- 
terdisciplinary units, and setting high and consistent expectations. 

• Friction has been reduced when it has been possible to select NGA 
teachers who enjoy working with ninth-graders and are open to work- 
ing in teams. Tensions have been evident when teachers have not had a 
voice in their assigmnents, especially when they do not understand the ra- 
tionale behind NGAs. In the District of Columbia, where teachers placed in 
the NGAs had no prior exposure to the concept, the rate of NGA teacher 
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turnover in the first year was 66 percent. On the other hand, teachers who 
have made a choice to be part of an NGA have generally been happy with 
this assignment. As one administrator commented, the teachers in the NGA 
at her school, most of whom volunteered to teach there, regard it as “Shangri- 
La.” She noted that the teachers enjoyed the teamwork; they also appreciated 
the NGA’s orderly, respectful environment. 

• The presence of a school administrator charged with overseeing the 
NGA has been important to smooth NGA functioning. Administrators — 
often assistant principals — have had broad oversight responsibilities for the 
NGAs. They attend the teacher meetings held during common planning time 
and help make sure that the time is used productively. 

• Most NGAs that were studied benefited from receiving technical assis- 
tance from an outside organization. The schools contracted with outside 
organizations to provide assistance on a variety of issues, from helping 
teachers work together in teams to enhancing instructional rigor and rele- 
vance. It may not be reasonable to expect school districts on their own to 
provide all the kinds of support that administrators and teachers may need to 
implement NGAs with maximum effectiveness. 

• Backing from the district or the superintendent, as well as from the 
school’s principal, has been critical for sustaining and strengthening the 
NGA over time. The NGA, like any innovation, benefits greatly from a 
champion who understands it and pushes it forward. NGAs that have contin- 
ued and thrived have generally been able to count on the support of the dis- 
trict administrator — especially critical after the initial grants expired and the 
additional costs associated with the NGA (common planning time for teach- 
ers, for example) have had to be supported through district funds. At the 
school level, supportive principals have helped ensure that the NGA’s staff- 
ing and scheduling needs have been met. 

• In explaining why the NGAs at their schools were disbanded, respond- 
ents cited conflicting priorities and a lack of continuing support from 
teachers and administrators, sometimes because of leadership turnover. 

Some schools eliminated their NGAs (or, in one case, never implemented an 
NGA in the first place) because district and school administrators saw them 
as too restrictive of students’ choices of courses and electives. At one school, 
teachers of upper-level classes and electives lobbied against the NGA be- 
cause they felt that it had negative effects on the master schedule. Then, too, 
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when new district administrators and others who had not been associated 
with the NGAs’ initial implementation took over, they often questioned the 
decisions of their predecessors. 

• Most schools with NGAs — including some in which the NGAs were 
discontinued after the initial funding expired — have seen improved 
student outcomes. Although those interviewed did not cite improvements in 
ninth-grade students’ academic achievement, they frequently noted that after 
NGAs were implemented, the students registered higher rates of promotion 
to tenth grade and lower rates of truancy and discipline problems. While bet- 
ter outcomes cannot, in and of themselves, be seen as an effect of the NGAs, 
the results of this supplementary study suggest that additional rigorous evalu- 
ation of the NGA concept may be warranted. 
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